Data preprocessing method for machine learning algorithm and related device

Abstract

The present disclosure provides a data preprocessing method for a machine learning algorithm and a related device. The method comprises: obtaining original data to be processed; screening based on missing values of each feature of the original data to obtain first screening data; screening based on identical values of each feature in the first screening data to determine second screening data; filling in missing values of each feature in the second screening data to obtain complete data; and performing standardization processing on the complete data according to a preset standardization processing rule to obtain the complete data after standardization processing. The embodiment of the present disclosure can preprocess structured data, improve the availability and quality of the data by processing the abnormal values of the data, and save a large amount of manpower and resources during the development of the machine learning algorithm.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data preprocessing method and related equipment for machine learning algorithms. Background Technology

[0002] With the development of machine learning technology, the demand for machine learning technology in more and more industries is becoming increasingly urgent. However, in the development of machine learning algorithms based on structured data, such as data mining and user profiling, a certain degree of data loss occurs during data collection. If machine learning algorithms are developed directly without filtering, it will lead to increased training time or decreased performance due to invalid data, or even failure to train when missing values ​​are present. Currently, missing value imputation is often done through brainstorming or consulting relevant experts, resulting in a significant waste of human and material resources. Existing technologies use Principal Component Analysis (PCA) dimensionality reduction and Convolutional Neural Networks (CNNs) to reduce the impact of invalid features on the results. However, the new features generated by PCA are difficult to establish with the actual application scenario, and the threshold of cumulative interpretable variance needs to be manually set and adjusted; the latter lacks sufficient biological support, has no memory function, and the fully connected mode of CNNs is too redundant and inefficient. In addition, both of the above methods have the problem that dimensionality reduction methods fail when the dataset contains missing values. Summary of the Invention

[0003] In view of this, the purpose of this disclosure is to propose a data preprocessing method and related equipment for machine learning algorithms.

[0004] For the purposes described above, this disclosure provides a data preprocessing method for machine learning algorithms, including:

[0005] Obtain the raw data to be processed;

[0006] Based on the missing values ​​of each feature in the original data, a first screening data is obtained;

[0007] The second screening data is determined by screening based on the same value of each feature in the first screening data;

[0008] The missing values ​​of each feature in the second screening data are filled in to obtain complete data;

[0009] The complete data is standardized according to preset standardization rules to obtain the standardized complete data.

[0010] Furthermore, the original data includes discrete data and continuous data, and both the discrete data and the continuous data include multiple features.

[0011] Furthermore, the screening based on the missing values ​​of each feature in the original data includes:

[0012] The number of missing values ​​for each feature in the original data is counted to determine the first proportion of the missing values ​​in the total data volume of the feature.

[0013] In response to determining that the first proportion is greater than a preset first threshold, the feature containing the missing value is deleted.

[0014] Furthermore, screening is performed based on the same value of each feature in the first screening data, including:

[0015] Perform the following operations on each feature of the discrete data in the first screening data:

[0016] Based on the number of each identical value, determine the second proportion of the identical value in the total amount of data in the feature containing the missing value;

[0017] In response to determining that the second proportion is greater than a preset second threshold, features containing the same value are deleted; otherwise...

[0018] Based on the number of any one identical value, determine the third proportion of that identical value in the total number of identical values;

[0019] In response to determining that the third proportion is greater than a preset third threshold, features containing the same value are deleted.

[0020] Furthermore, screening based on the same value of each feature in the first screening data also includes:

[0021] Perform the following operations on each feature of the continuous data in the first screening data:

[0022] Based on the number of each identical value, determine the fourth proportion of the identical value in the total amount of data containing the missing value in the feature to which it belongs;

[0023] In response to determining that the fourth proportion is greater than a preset fourth threshold, features containing the same value are deleted.

[0024] Furthermore, the step of filling in the missing values ​​of each feature in the second screening data includes:

[0025] For each feature of the discrete data in the second screening data, perform the following operations:

[0026] Based on the number of missing values, determine the fifth proportion of the missing values ​​in the total data volume of the corresponding feature;

[0027] In response to determining that the fifth proportion is less than a preset fifth threshold, data is randomly sampled based on the frequency of each value to fill in the missing values; otherwise...

[0028] A classification model is built using support vector machines based on data that does not contain the missing values.

[0029] The missing values ​​are filled in according to the pre-trained classification model.

[0030] Furthermore, the step of filling in the missing values ​​of each feature in the second screening data also includes:

[0031] For each feature of the continuous data in the second screening data, perform the following operations:

[0032] Based on the number of missing values, determine the sixth proportion of the missing values ​​in the total data volume of the corresponding feature;

[0033] In response to determining that the sixth proportion is less than a preset sixth threshold, the missing values ​​are filled using a Gaussian distribution based on the maximum, minimum, and median of the non-missing value data; otherwise...

[0034] Based on data that does not contain the missing values, a regression model is established using multinomial regression.

[0035] The missing values ​​are filled in according to the pre-trained regression model.

[0036] Furthermore, the preset standardization rules include standardization criteria for discrete data and standardization criteria for continuous data, wherein,

[0037] The standardization processing rules for the discrete data include One-Hot encoding;

[0038] The standardization rules for continuous data include min-max standardization, z-score standardization, and logarithmic function transformation.

[0039] Based on the same inventive concept, this disclosure also provides a data preprocessing apparatus for machine learning algorithms, comprising:

[0040] Acquisition module: Configured to acquire the raw data to be processed;

[0041] The first screening module is configured to screen based on the missing values ​​of each feature of the original data to obtain the first screening data.

[0042] The second screening module is configured to screen based on the same value of each feature in the first screening data to determine the second screening data.

[0043] Fill module: configured to fill in the missing values ​​of each feature in the second screening data to obtain complete data;

[0044] Standardization module: configured to standardize the complete data according to preset standardization rules to obtain the standardized complete data.

[0045] Based on the same inventive concept, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above.

[0046] As can be seen from the above, the data preprocessing method and related equipment for machine learning algorithms provided in this disclosure, after acquiring the raw data to be processed, screens out abnormal data in the raw data, including missing values ​​and redundant identical values, and removes the abnormal data, reducing the time required for actual data screening and improving data usability. In addition, missing values ​​in the screened data are filled in, which improves data quality while ensuring data integrity. Finally, the data is standardized, and the standardized data improves the accuracy and efficiency of model training and data analysis, saving a lot of manpower and resources. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart of a data preprocessing method for machine learning algorithms according to an embodiment of the present disclosure;

[0049] Figure 2 This is a schematic diagram of the data preprocessing apparatus for machine learning algorithms according to an embodiment of the present disclosure;

[0050] Figure 3 This is a schematic diagram of the electronic device structure according to an embodiment of the present disclosure. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0052] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Words such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, but do not exclude other elements or objects.

[0053] As described in the background section, the data preprocessing techniques used in related technologies for machine learning algorithms generally process structured data using principal component analysis (PCA) dimensionality reduction or convolutional neural networks. However, in implementing this disclosure, the applicant discovered the following problems with data preprocessing using PCA dimensionality reduction or convolutional neural networks:

[0054] The new features generated by principal component analysis dimensionality reduction are unrelated to each other. Therefore, in some cases, the new features cannot be linked to the actual application scenario. In addition, it is necessary to manually set and adjust the threshold of cumulative interpretable variance.

[0055] Convolutional neural networks lack sufficient biological support, have no memory function, and their fully connected mode is overly redundant and inefficient.

[0056] In addition, both of the above methods suffer from the problem that dimensionality reduction methods fail when there are missing values ​​in the dataset, wasting a lot of human and material resources.

[0057] In view of this, embodiments of the present disclosure provide a data preprocessing method for machine learning algorithms, which can preprocess structured data and improve the defects caused by missing values ​​and identical values ​​in data features, thereby improving the quality and efficiency of machine learning algorithm development.

[0058] The technical solutions of this disclosure will be described in detail below through specific embodiments.

[0059] refer to Figure 1 The data preprocessing method for machine learning algorithms according to embodiments of this disclosure includes the following steps:

[0060] Step S101: Obtain the raw data to be processed.

[0061] In this step, the original data includes discrete and continuous data. Both discrete and continuous data contain multiple features, and there is no logical relationship between any two features, meaning that changes in the position of a feature will not affect the data. Different features are distinguished by setting a corresponding feature name for each feature.

[0062] Specifically, continuous data is characterized by values ​​appearing within the same feature being much smaller than the total data volume, and these values ​​are finite or presented in textual form; continuous data is also characterized by data values ​​within the same feature being close to the total data volume, and all of them being numbers.

[0063] Step S102: Screen based on the missing values ​​of each feature of the original data to obtain the first screening data.

[0064] In this step, the raw data undergoes preliminary screening, specifically including:

[0065] The number of missing values ​​for each feature in the original data is counted to determine the first proportion of the missing values ​​in the total data volume of the feature.

[0066] In response to determining that the first proportion is greater than a preset first threshold, the feature containing the missing value is deleted.

[0067] Specifically, for example, if the proportion of missing values ​​in the total data volume of the feature exceeds 50%, the feature is deleted from the original data; otherwise, no operation is performed on the original data, and it is screened.

[0068] Step S103: Screen based on the same value of each feature in the first screening data to determine the second screening data.

[0069] In this step, it is necessary to screen the discrete and continuous data in the original data separately. The screening of the discrete data includes:

[0070] Perform the following operations on each feature of the discrete data in the first screening data:

[0071] Based on the number of each identical value, determine the second proportion of the identical value in the total amount of data in the feature containing the missing value;

[0072] In response to determining that the second proportion is greater than a preset second threshold, features containing the same value are deleted; otherwise...

[0073] Based on the number of any one identical value, determine the third proportion of that identical value in the total number of identical values;

[0074] In response to determining that the third proportion is greater than a preset third threshold, features containing the same value are deleted.

[0075] Specifically, for example, if any identical value in the discrete data accounts for more than 90% of the total data volume (including missing values) of its respective feature, the feature containing that identical value will be deleted; otherwise, if any identical value accounts for more than 90% of the total data volume of all identical values, the feature containing that identical value will be deleted; if it does not exceed 90%, the feature passes the screening in this step.

[0076] Further screening of continuous data includes:

[0077] Perform the following operations on each feature of the continuous data in the first screening data:

[0078] Based on the number of each identical value, determine the fourth proportion of the identical value in the total amount of data containing the missing value in the feature to which it belongs;

[0079] In response to determining that the fourth proportion is greater than a preset fourth threshold, features containing the same value are deleted.

[0080] Specifically, for example, if the proportion of any identical value in the total amount of data (including missing values) of its corresponding feature in continuous data is greater than 10%, the feature containing that identical value will be deleted; otherwise, it will pass the screening in this step.

[0081] Step S104: Fill in the missing values ​​of each feature in the second screening data to obtain complete data.

[0082] In this step, it is also necessary to fill in the missing values ​​of each feature in both the discrete and continuous data of the second screening data. Filling in the missing values ​​of each feature in the discrete data includes:

[0083] For each feature of the discrete data in the second screening data, perform the following operations:

[0084] Based on the number of missing values, determine the fifth proportion of the missing values ​​in the total data volume of the corresponding feature;

[0085] In response to determining that the fifth proportion is less than a preset fifth threshold, data is randomly sampled based on the frequency of each value to fill in the missing values; otherwise...

[0086] A classification model is built using support vector machine (SVM) based on data that does not contain the missing values.

[0087] The missing values ​​are filled in according to the pre-trained classification model.

[0088] Specifically, for example, when the proportion of missing values ​​in the total data volume of the feature in the discrete data is less than 30%, data is randomly sampled according to the frequency of each value in this feature to fill in the missing values; otherwise, a classification model is built and trained using SVM based on the data in this feature that does not contain missing values, and the missing values ​​are filled in based on the trained classification model.

[0089] Furthermore, imputing missing values ​​for each feature in continuous data includes:

[0090] For each feature of the continuous data in the second screening data, perform the following operations:

[0091] Based on the number of missing values, determine the sixth proportion of the missing values ​​in the total data volume of the corresponding feature;

[0092] In response to determining that the sixth proportion is less than a preset sixth threshold, the missing values ​​are filled using a Gaussian distribution based on the maximum, minimum, and median of the non-missing value data; otherwise...

[0093] Based on data that does not contain the missing values, a regression model is established using multinomial regression.

[0094] The missing values ​​are filled in according to the pre-trained regression model.

[0095] Specifically, for example, when the proportion of missing values ​​in the total data volume of a continuous feature is less than 30%, the missing values ​​are filled using a Gaussian distribution based on the maximum, minimum, and median of the non-missing values ​​in this feature. In the Gaussian distribution used for missing value filling, the median represents the expectation of the Gaussian distribution, and half the difference between the maximum and minimum values ​​represents the variance of the Gaussian distribution. When the proportion of missing values ​​in the total data volume of a continuous feature is not less than 30%, a regression model is built and trained using multinomial regression based on the non-missing values ​​in this feature, and the missing values ​​are filled based on the trained regression model.

[0096] Step S105: Standardize the complete data according to the preset standardization processing rules to obtain the standardized complete data.

[0097] In this step, the preset standardization rules include standardization criteria for discrete data and standardization criteria for continuous data. Specifically, the standardization rules for continuous data include min-max standardization, z-score standardization, and logarithmic function transformation.

[0098] The formula for calculating min-max normalization is as follows:

[0099]

[0100] In the formula, X represents the feature vector, x represents the unprocessed initial data, and x'∈X represents a single normalized data point.

[0101] The formula for calculating the z-score standardization is as follows:

[0102]

[0103] In the formula, X * σ represents a single data point after normalization, μ represents the mean of all data points, and σ represents the variance of all data points.

[0104] The formula for calculating the logarithmic function transformation is as follows:

[0105]

[0106] In the formula, X′ represents a single data point after normalization, and max′ represents the maximum value among the features.

[0107] Furthermore, the standardization rule for discrete data is one-hot encoding.

[0108] Therefore, by processing the missing and identical values ​​in the original data, the final complete data can avoid the adverse effects of abnormal data in the original data on the accuracy and efficiency of machine learning model training and data analysis, thus ensuring that the output of the machine learning model has a higher confidence level.

[0109] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0110] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0111] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides a data preprocessing apparatus for machine learning algorithms.

[0112] refer to Figure 2 The data preprocessing device for machine learning algorithms includes:

[0113] Acquisition module 201: configured to acquire raw data to be processed;

[0114] First screening module 202: configured to screen based on the missing values ​​of each feature of the original data to obtain first screening data;

[0115] The second screening module 203 is configured to screen based on the same value of each feature in the first screening data to determine the second screening data;

[0116] Filling module 204: configured to fill in the missing values ​​of each feature in the second screening data to obtain complete data;

[0117] Standardization module 205: is configured to standardize the complete data according to preset standardization processing rules to obtain the standardized complete data.

[0118] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0119] The apparatus of the above embodiments is used to implement the corresponding data preprocessing method for machine learning algorithms in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0120] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data preprocessing method for machine learning algorithms described in any of the above embodiments.

[0121] Figure 3 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0122] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0123] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0124] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0125] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0126] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0127] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0128] The electronic devices described above are used to implement the corresponding data preprocessing methods for machine learning algorithms in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0129] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the data preprocessing method for machine learning algorithms as described in any of the above embodiments.

[0130] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0131] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the data preprocessing method for machine learning algorithms as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0132] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0133] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0134] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0135] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data preprocessing method for a machine learning algorithm, comprising: obtaining original data to be processed; screening based on missing values of each feature of the original data to obtain first screening data; wherein the original data comprises discrete data and continuous data, and the discrete data and the continuous data each comprise a plurality of features; screening based on identical values of each feature in the first screening data to determine second screening data; wherein the following operations are performed on each feature of the discrete data in the first screening data: determining a second proportion of the identical value in the total data amount of the feature in which the identical value is contained based on the number of each identical value; in response to determining that the second proportion is greater than a preset second threshold, deleting the feature containing the identical value; otherwise determining a third proportion of the identical value in the total data amount of all identical values based on the number of any one identical value; in response to determining that the third proportion is greater than a preset third threshold, deleting the feature containing the identical value; filling missing values of each feature in the second screening data to obtain complete data; wherein the following operations are performed on each feature of the discrete data in the second screening data: determining a fifth proportion of the missing value in the total data amount of the feature based on the number of missing values; in response to determining that the fifth proportion is less than a preset fifth threshold, filling the missing values by randomly extracting data according to the frequency of occurrence of each value; otherwise establishing a classification model by a support vector machine based on data not containing the missing values; filling the missing values according to the pre-trained classification model; wherein the following operations are performed on each feature of the continuous data in the second screening data: determining a sixth proportion of the missing value in the total data amount of the feature based on the number of missing values; in response to determining that the sixth proportion is less than a preset sixth threshold, filling the missing values by a Gaussian distribution based on the maximum value, the minimum value, and the median of the non-missing value data; otherwise establishing a regression model by a polynomial regression based on data not containing the missing values; filling the missing values according to the pre-trained regression model; standardizing the complete data according to a preset standardization processing rule to obtain the standardized complete data.

2. The method of claim 1, wherein, The screening based on the missing values of each feature of the original data comprises: counting the number of missing values of each feature in the original data to determine a first proportion of the missing values in the total data amount of the feature; in response to determining that the first proportion is greater than a preset first threshold, deleting the feature containing the missing values.

3. The method of claim 2, wherein, The screening based on the identical values of each feature in the first screening data further comprises: the following operations are performed on each feature of the continuous data in the first screening data: determining a fourth proportion of the identical value in the total data amount of the feature in which the identical value is contained based on the number of each identical value; in response to determining that the fourth proportion is greater than a preset fourth threshold, deleting the feature containing the identical value.

4. The method of claim 1, wherein, The preset standardization processing rule includes a standardization processing criterion for discrete data and a standardization processing criterion for continuous data, wherein The standardization processing rule for the discrete data includes One-Hot encoding; The standardization processing rule for the continuous data includes deviation min-max standardization, z-score standardization, and logarithmic function conversion.

5. A data preprocessing apparatus for a machine learning algorithm, comprising: an acquisition module configured to acquire original data to be processed; a first screening module configured to screen based on missing values of each feature of the original data to obtain first screened data; wherein the original data includes discrete data and continuous data, and the discrete data and the continuous data each include a plurality of features; a second screening module configured to screen based on identical values of each feature in the first screened data to determine second screened data; wherein for each feature of the discrete data in the first screened data, the following operations are performed: based on a number of each identical value, a second proportion of the identical value in a total amount of data in which the identical value belongs to the feature is determined; in response to determining that the second proportion is greater than a preset second threshold, the feature containing the identical value is deleted; otherwise based on a number of any one identical value, a third proportion of the identical value in a total amount of all identical values is determined; in response to determining that the third proportion is greater than a preset third threshold, the feature containing the identical value is deleted; a filling module configured to fill missing values of each feature in the second screened data to obtain complete data; wherein for each feature of the discrete data in the second screened data, the following operations are performed: based on a number of missing values, a fifth proportion of the missing values in a total amount of data of the feature to which the missing values belong is determined; in response to determining that the fifth proportion is less than a preset fifth threshold, the missing values are filled by randomly extracting data according to a frequency of occurrence of each value; otherwise a classification model is established by a support vector machine based on data not containing the missing values; the missing values are filled according to the pre-trained classification model; wherein for each feature of the continuous data in the second screened data, the following operations are performed: based on a number of missing values, a sixth proportion of the missing values in a total amount of data of the feature to which the missing values belong is determined; in response to determining that the sixth proportion is less than a preset sixth threshold, the missing values are filled by a Gaussian distribution based on a maximum value, a minimum value, and a median of non-missing value data; otherwise a regression model is established by a polynomial regression based on data not containing the missing values; the missing values are filled according to the pre-trained regression model; a standardization module configured to standardize the complete data according to a preset standardization processing rule to obtain the complete data after standardization processing.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 4 when executing the program.

Images