Method for processing null value and abnormal value of telecom operator user form data

Abstract

The invention discloses a telecom operator user form data null value and abnormal value processing method, which belongs to the technical field of data mining, and comprises the following steps: S1, screening out high-quality data without missing values and abnormal values in original data as an initial data set 1 (assuming that the initial data proof number is n); through a series of steps, the data quality is remarkably improved, and data resources can be fully utilized, so that the accuracy of data analysis is ensured; by screening high-quality data, removing invalid samples, accurately predicting null values, adopting a comprehensive filling strategy, effectively identifying and processing abnormal values, optimizing model performance, keeping flexibility and expandability, gradually perfecting a data set, adjusting hyper-parameters and utilizing professional knowledge, comprehensive optimization of the data set is achieved, and the method is suitable for large-scale popularization and application. According to the method, the overall availability and reliability of the data set are improved, and powerful support is provided for data analysis and business decision making by combining a machine learning algorithm and personnel experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of data mining, specifically to the data preprocessing link in the data mining field, and in particular to a method for processing null values ​​and abnormal values ​​in user form data of telecommunications operators. Background Art

[0002] When telecom operators conduct user data mining and analysis, the data resources obtained from the database generally have many missing values ​​and outliers. At present, in the process of data preprocessing, telecom operators basically use the mean, median, mode and other numerical values ​​to directly fill in, or use regression algorithms to predict data. From the perspective of form data, using specific numerical filling such as mean and median only considers the characteristics of a single feature, while using regression algorithms only uses different features in a single sample for prediction, which does not give full play to the role of all valid data in the form.

[0003] At the same time, domestic and foreign scholars have proposed many methods for filling missing values ​​and identifying outliers. These methods have played a huge role in different fields. However, the processing of empty values ​​and outliers in form data is basically achieved with the help of specific value filling and regression algorithm prediction. The method of raw data preprocessing is similar to the data processing method of telecom operators.

[0004] The embodiment of the present invention provides a data processing method, which mainly focuses on the processing of null values ​​and outliers to solve the problem of insufficient utilization of original data resources in the process of processing null values ​​and outliers in the prior art, especially in the regression algorithm. Summary of the invention

[0005] The purpose of the present invention is to make full use of the limited valid data in the original data, adopt the weighted average of the regression model prediction value and the specific numerical value, and add the pre-processed data value to the model training data, so as to solve the problem that there is less high-quality data in the original data, the model training effect is limited, and the accuracy of null value and outlier identification and filling is low.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: A method for processing null values ​​and abnormal values ​​in user form data of a telecommunications operator, comprising: S1: Filter out high-quality data without missing values ​​and outliers from the original data as the initial data set 1 (assuming that the number of initial data features is n); S2: Eliminate samples in the existing data where the number of characteristic null values ​​in a single sample is greater than a specific value; S3: Combined with the initial data set 1, with the help of machine learning algorithms, a prediction model with one missing value is trained, and the performance of the prediction model is evaluated. Several machine learning models are comprehensively compared to obtain the prediction model with the best statistical performance; S4: Filter out the samples with 1 characteristic null value in the original data, and use the prediction model trained in step S3 to predict the null value. Perform weighted average on the obtained prediction value, data mean, data mode, data median, and personnel experience value to obtain the final null value filling value, and fill it; S5: The initial data and the data after the number of empty values ​​of the sample features is padded to 1 form a new data set. With the help of the prediction model trained in step S3, each feature of each sample in the new data set is predicted in a traversal manner to obtain the predicted value. The predicted value is weighted averaged with the data mean, data mode, data median, and personnel experience value to obtain the final filling value. The abnormal value filling value is compared with the actual value in the new data set. When the difference between the predicted value and the filling value is within the set error range, the actual value is retained. When the difference between the filling value and the actual value is not within the set error range, the actual value is replaced with the predicted value. Then, a data set with the initial data and the number of empty values ​​of the sample features padded to 1 and processed for abnormal values ​​is obtained. This data set is used as the initial data set 2 for the following steps; S6: Combined with the latest initial data set 2, with the help of machine learning algorithms, a prediction model with two missing values ​​is trained, and the performance of the prediction model is evaluated to obtain the prediction model with the best statistical performance; S7: Combined with the latest initial data set 2, with the help of machine learning algorithms, a prediction model with 1 missing value is trained, and the performance of the prediction model is evaluated to obtain the prediction model with the best statistical performance; S8: Filter out samples with 2 characteristic null values ​​in the original data, and use the prediction model trained in step S6 to predict null values. Perform weighted average on the obtained prediction value, data mean, data mode, data median, and personnel experience value to obtain the final null value filling value, and fill it; S9: The latest initial data and the data after the sample feature null value number is padded to 2 form a new data set. With the help of the prediction model trained in step S7, a traversal method is adopted to predict the predicted value of each feature of each sample in the new data set. The obtained predicted value is weighted averaged with the data mean, data mode, data median, and personnel experience value to obtain the final outlier filling value. The filling value is compared with the actual value in the new data set. When the difference between the filling value and the actual value is within the set error range, the actual value is retained. When the difference between the filling value and the actual value is not within the set error range, the actual value is replaced with the filling value. Then the latest initial data, the sample feature null value number is padded to 2 and the data processed for outliers are obtained to form a new data set. And continue to use this data set as the initial data set 3 in the following steps; S10: Continue to repeat the above steps until a complete data set without null values ​​or outliers is obtained.

[0007] As a further description of the above technical solution: In step S1, the screening of high-quality data is achieved by first excluding data containing missing values ​​and then screening and eliminating outliers in turn; alternatively, data with more missing values ​​and outliers can be eliminated, and data with fewer missing values ​​and outliers can be improved by using manual experience combined with existing mean filling, median filling, maximum likelihood estimation function filling and other methods.

[0008] As a further description of the above technical solution: In step S2, samples in the existing data whose number of characteristic null values ​​of a single sample exceeds a specific value are eliminated, where the specific value can be a value greater than (n / 2-1) rounded to an integer, or a value determined by the data processing personnel based on experience, in order to exclude data with too many missing values ​​and no actual repair value.

[0009] As a further description of the above technical solution: In the steps S3, S6 and S7, the machine learning algorithms used include but are not limited to one or more of K nearest neighbor, decision tree, random forest, ensemble algorithm, etc. During the training of the prediction model, the initial data is divided into a training set and a test set, and the above algorithms are used to train prediction models for a specific number of missing values.

[0010] As a further description of the above technical solution: In the steps S3, S6 and S7, the performance evaluation of the prediction model adopts one or more methods of precision, recall and ROC curve, and the priority of these statistical characteristics is set according to the specific situation of the data.

[0011] As a further description of the above technical solution: In the steps S3 and S6, the number of training models is determined as follows: when the number of null values ​​to be processed is 1, the number of models is C(n,1); when the number of null values ​​to be processed is 2, the number of models is C(n,2); and so on.

[0012] As a further description of the above technical solution: In step S4, the final null value filling value is the weighted average result of the predicted value, feature mean, feature mode, feature median, and personnel experience value of the machine learning algorithm training model, among which the weights of the predicted value, feature mean, feature mode, feature median, and personnel experience value in the weighted average process are used as hyperparameters and can be adjusted in actual applications.

[0013] As a further description of the above technical solution: In step S4, the personnel experience value is a value set by personnel who are familiar with the data based on historical experience.

[0014] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: The present invention significantly improves data quality through a series of carefully designed steps, and can make full use of data resources, thereby ensuring the accuracy of data analysis. It achieves comprehensive optimization of the data set by screening high-quality data, eliminating invalid samples, accurately predicting null values, adopting comprehensive filling strategies, effectively identifying and processing outliers, optimizing model performance, maintaining flexibility and scalability, gradually improving the data set, adjusting hyperparameters, and utilizing professional knowledge. This method not only improves the overall availability and reliability of the data set, but also provides strong support for data analysis and business decision-making by combining machine learning algorithms and human experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Detailed schematic diagram of the overall steps of the present invention Figure 2 Schematic diagram of the empty value filling value confirmation module of the present invention Figure 3 Schematic diagram of the abnormal value filling value confirmation module of the present invention Figure 4 Schematic diagram of an abnormal value filling value identification and filling module of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Please see attached Figure 1 -Attached Figure 4 The present invention provides a technical solution: a method for processing null values ​​and abnormal values ​​of user form data of a telecommunications operator, comprising: S1: Filter out high-quality data without missing values ​​and outliers from the original data as the initial data set 1 (assuming that the number of initial data features is n); S2: Eliminate samples in the existing data where the number of characteristic null values ​​in a single sample is greater than a specific value; S3: Combined with the initial data set 1, with the help of machine learning algorithms, a prediction model with one missing value is trained, and the performance of the prediction model is evaluated. Several machine learning models are comprehensively compared to obtain the prediction model with the best statistical performance; S4: Filter out the samples with 1 characteristic null value in the original data, and use the prediction model trained in step S3 to predict the null value. Perform weighted average on the obtained prediction value, data mean, data mode, data median, and personnel experience value to obtain the final null value filling value, and fill it; S5: The initial data and the data after the number of empty values ​​of the sample features is padded to 1 form a new data set. With the help of the prediction model trained in step S3, each feature of each sample in the new data set is predicted in a traversal manner to obtain the predicted value. The predicted value is weighted averaged with the data mean, data mode, data median, and personnel experience value to obtain the final filling value. The abnormal value filling value is compared with the actual value in the new data set. When the difference between the predicted value and the filling value is within the set error range, the actual value is retained. When the difference between the filling value and the actual value is not within the set error range, the actual value is replaced with the predicted value. Then, a data set with the initial data and the number of empty values ​​of the sample features padded to 1 and processed for abnormal values ​​is obtained. This data set is used as the initial data set 2 for the following steps; S6: Combined with the latest initial data set 2, with the help of machine learning algorithms, a prediction model with two missing values ​​is trained, and the performance of the prediction model is evaluated to obtain the prediction model with the best statistical performance; S7: Combined with the latest initial data set 2, with the help of machine learning algorithms, a prediction model with 1 missing value is trained, and the performance of the prediction model is evaluated to obtain the prediction model with the best statistical performance; S8: Filter out samples with 2 characteristic null values ​​in the original data, and use the prediction model trained in step S6 to predict null values. Perform weighted average on the obtained prediction value, data mean, data mode, data median, and personnel experience value to obtain the final null value filling value, and fill it; S9: The latest initial data and the data after the sample feature null value number is padded to 2 form a new data set. With the help of the prediction model trained in step S7, a traversal method is adopted to predict the predicted value of each feature of each sample in the new data set. The obtained predicted value is weighted averaged with the data mean, data mode, data median, and personnel experience value to obtain the final outlier filling value. The filling value is compared with the actual value in the new data set. When the difference between the filling value and the actual value is within the set error range, the actual value is retained. When the difference between the filling value and the actual value is not within the set error range, the actual value is replaced with the filling value. Then the latest initial data, the sample feature null value number is padded to 2 and the data processed for outliers are obtained to form a new data set. And continue to use this data set as the initial data set 3 in the following steps; S10: Continue to repeat the above steps until a complete data set without null values ​​or outliers is obtained.

[0018] In step S1, the screening of high-quality data is achieved by first excluding data with missing values ​​and then screening and eliminating outliers in turn; alternatively, data with more missing values ​​and outliers can be eliminated, and for data with fewer missing values ​​and outliers, manual experience combined with existing mean filling, median filling, maximum likelihood estimation function filling and other methods can be used to improve them.

[0019] In step S2, samples whose number of characteristic null values ​​of a single sample in the existing data exceeds a specific value are eliminated, where the specific value can be a value greater than (n / 2-1) rounded to an integer, or a value determined by the data processing personnel based on experience, in order to exclude data with too many missing values ​​and no practical repair value.

[0020] In steps S3, S6, and S7, the machine learning algorithms used include, but are not limited to, one or more of K-nearest neighbor, decision tree, random forest, and ensemble algorithms. During the training of the prediction model, the initial data is divided into a training set and a test set, and the above algorithms are used to train prediction models for a specific number of missing values.

[0021] In steps S3, S6, and S7, the performance evaluation of the prediction model adopts one or more methods of precision, recall, and ROC curve, and the priority of these statistical characteristics is set according to the specific situation of the data.

[0022] In steps S3 and S6, the number of training models is determined as follows: when the number of null values ​​processed is 1, the number of models is C(n,1); when the number of null values ​​processed is 2, the number of models is C(n,2); and so on.

[0023] In step S4, the final null value filling value is the weighted average result of the predicted value, feature mean, feature mode, feature median, and personnel experience value of the machine learning algorithm training model, among which the weights of the predicted value, feature mean, feature mode, feature median, and personnel experience value in the weighted average process are used as hyperparameters and can be adjusted in actual applications.

[0024] In step S4, the personnel experience value is a value set by personnel who are familiar with the data based on historical experience.

[0025] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for processing null values ​​and abnormal values ​​in user form data of a telecommunications operator, characterized in that: include: S1: Filter out high-quality data without missing values ​​and outliers from the original data as the initial data set 1 (assuming that the number of initial data features is n); S2: Eliminate samples in the existing data where the number of characteristic null values ​​in a single sample is greater than a specific value; S3: Combined with the initial data set 1, with the help of machine learning algorithms, a prediction model with one missing value is trained, and the performance of the prediction model is evaluated. Several machine learning models are comprehensively compared to obtain the prediction model with the best statistical performance; S4: Filter out the samples with 1 characteristic null value in the original data, and use the prediction model trained in step S3 to predict the null value. Perform weighted average on the obtained prediction value, data mean, data mode, data median, and personnel experience value to obtain the final null value filling value, and fill it; S5: The initial data and the data after the number of empty values ​​of the sample features is padded to 1 form a new data set. With the help of the prediction model trained in step S3, each feature of each sample in the new data set is predicted in a traversal manner to obtain the predicted value. The predicted value is weighted averaged with the data mean, data mode, data median, and personnel experience value to obtain the final filling value. The abnormal value filling value is compared with the actual value in the new data set. When the difference between the predicted value and the filling value is within the set error range, the actual value is retained. When the difference between the filling value and the actual value is not within the set error range, the actual value is replaced with the predicted value. Then, a data set with the initial data and the number of empty values ​​of the sample features padded to 1 and processed for abnormal values ​​is obtained. This data set is used as the initial data set 2 for the following steps; S6: Combined with the latest initial data set 2, with the help of machine learning algorithms, a prediction model with two missing values ​​is trained, and the performance of the prediction model is evaluated to obtain the prediction model with the best statistical performance; S7: Combined with the latest initial data set 2, with the help of machine learning algorithms, a prediction model with 1 missing value is trained, and the performance of the prediction model is evaluated to obtain the prediction model with the best statistical performance; S8: Filter out samples with 2 characteristic null values ​​in the original data, and use the prediction model trained in step S6 to predict null values. Perform weighted average on the obtained prediction value, data mean, data mode, data median, and personnel experience value to obtain the final null value filling value, and fill it; S9: The latest initial data and the data after the sample feature null value number is padded to 2 form a new data set. With the help of the prediction model trained in step S7, a traversal method is adopted to predict the predicted value of each feature of each sample in the new data set. The obtained predicted value is weighted averaged with the data mean, data mode, data median, and personnel experience value to obtain the final outlier filling value. The filling value is compared with the actual value in the new data set. When the difference between the filling value and the actual value is within the set error range, the actual value is retained. When the difference between the filling value and the actual value is not within the set error range, the actual value is replaced with the filling value. Then the latest initial data, the sample feature null value number is padded to 2 and the data processed for outliers are obtained to form a new data set. And continue to use this data set as the initial data set 3 in the following steps; S10: Continue to repeat the above steps until a complete data set without null values ​​or outliers is obtained.

2. A method for processing null values ​​and abnormal values ​​of user form data of a telecommunications operator according to claim 1, characterized in that: In step S1, the screening of high-quality data is achieved by first excluding data containing missing values ​​and then screening and eliminating outliers in turn; alternatively, data with more missing values ​​and outliers can be eliminated, and data with fewer missing values ​​and outliers can be improved by using manual experience combined with existing mean filling, median filling, maximum likelihood estimation function filling and other methods.

3. A method for processing null values ​​and abnormal values ​​of user form data of a telecommunications operator according to claim 1, characterized in that: In step S2, samples in the existing data whose number of characteristic null values ​​of a single sample exceeds a specific value are eliminated, where the specific value can be a value greater than (n / 2-1) rounded to an integer, or a value determined by the data processing personnel based on experience, in order to exclude data with too many missing values ​​and no actual repair value.

4. A method for processing null values ​​and abnormal values ​​of user form data of a telecommunications operator according to claim 1, characterized in that: In the steps S3, S6 and S7, the machine learning algorithms used include but are not limited to one or more of K nearest neighbor, decision tree, random forest, ensemble algorithm, etc. During the training of the prediction model, the initial data is divided into a training set and a test set, and the above algorithms are used to train prediction models for a specific number of missing values.

5. A method for processing null values ​​and abnormal values ​​of user form data of a telecommunications operator according to claim 1, characterized in that: In the steps S3, S6 and S7, the performance evaluation of the prediction model adopts one or more methods of precision, recall and ROC curve, and the priority of these statistical characteristics is set according to the specific situation of the data.

6. A method for processing null values ​​and abnormal values ​​of user form data of a telecommunications operator according to claim 1, characterized in that: In the steps S3 and S6, the number of training models is determined as follows: when the number of null values ​​to be processed is 1, the number of models is C(n,1); when the number of null values ​​to be processed is 2, the number of models is C(n,2); and so on.

7. A method for processing null values ​​and abnormal values ​​of user form data of a telecommunications operator according to claim 1, characterized in that: In step S4, the final null value filling value is the weighted average result of the predicted value, feature mean, feature mode, feature median, and personnel experience value of the machine learning algorithm training model, among which the weights of the predicted value, feature mean, feature mode, feature median, and personnel experience value in the weighted average process are used as hyperparameters and can be adjusted in actual applications.

8. A method for processing null values ​​and abnormal values ​​of user form data of a telecommunications operator according to claim 1, characterized in that: In step S4, the personnel experience value is a value set by personnel who are familiar with the data based on historical experience.

Images