Power multi-source data processing method and device, computer equipment, readable storage medium and program product

By obtaining the field characteristics of multi-source data and using machine learning models to process abnormal data, the problem of insufficient flexibility in multi-source data preprocessing is solved, and the intelligent and automated processing of multi-source data is realized, and the accuracy of preprocessing is improved. Sex and flexibility.

CN118916620APending Publication Date: 2024-11-08南方电网能源发展研究院有限责任公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410962811.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art is difficult to effectively process dynamics and uncertainties in multi-source data, resulting in a fixed rule data preprocessing method that lacks flexibility and cannot accurately and effectively process multi-source data.

Method used

By obtaining the field characteristics of the data to be processed, identifying the type of abnormal data, and determining the corresponding missing or out-of-bounds processing schemes based on the characteristics, using machine learning models for classification processing, adaptively determining the processing method, and improving the intelligence of multi-source data and Automated processing capability.

Benefits of technology

It realizes accurate and effective preprocessing of multi-source data, improves the flexibility and accuracy of data processing, and adapts to data processing needs of different exception types.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916620A_ABST
    Figure CN118916620A_ABST
Patent Text Reader

Abstract

The invention relates to an electric power multi-source data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining an exception type of exceptional data in data to be processed, determining a missing processing scheme corresponding to the exceptional data according to field features under the condition that the exception type is missing exception, and determining a border crossing processing scheme corresponding to the exceptional data according to the field features under the condition that the exception type is border crossing exception, and processing the abnormal data through a corresponding processing scheme. By adopting the method, the processing mode can be adaptively determined for the abnormal data of different abnormal types, intelligent and automatic processing of the abnormal data is achieved, the flexibility of data preprocessing of the multi-source data is improved, and therefore the multi-source data can be accurately and effectively preprocessed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment, readable storage medium and program product for processing multi-source power data. Background Art

[0002] With the advent of the big data era, data has become an important asset and source of competitiveness for enterprises. In the current business environment, enterprises need to use efficient data analysis to gain market insights, optimize business processes, and improve decision-making efficiency. With the development of information technology, the data sources of enterprises have also become diversified, forming multi-source data.

[0003] At present, the preprocessing methods for multi-source data are mostly based on fixed rules. For data from different sources, corresponding data filtering and data cleaning rules are set for data preprocessing. Since the generation of multi-source data is usually dynamic and uncertain, and the source of data is also dynamically changing, the data preprocessing method with fixed rules lacks flexibility in practical applications and is difficult to accurately and effectively preprocess multi-source data. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, readable storage medium and program product for processing multi-source power data that can improve the accuracy of multi-source data processing in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for processing multi-source power data, comprising:

[0006] Acquire field characteristics of the data to be processed; the field characteristics include the abnormal type of abnormal data in the data to be processed;

[0007] When the exception type is a missing exception, determine the missing processing solution corresponding to the abnormal data based on the field characteristics;

[0008] When the exception type is an out-of-bounds exception, determine the out-of-bounds processing solution corresponding to the abnormal data based on the field characteristics;

[0009] The abnormal data is processed through corresponding processing schemes.

[0010] In one embodiment, the field characteristics also include the field type of the data to be processed; the step of obtaining the field characteristics of the data to be processed includes:

[0011] Identify abnormal data and abnormal types of abnormal data in the data to be processed;

[0012] Perform field statistics on the data to be processed and obtain the field type of the data to be processed.

[0013] In one embodiment, the field feature further includes an upper quartile and a lower quartile; and the step of identifying abnormal data and abnormal types of abnormal data in the data to be processed includes:

[0014] The upper quartile and the lower quartile are used as boundary values, and the target distance between each data point in the data to be processed and the boundary value is calculated;

[0015] An out-of-bounds threshold is determined according to the target distance, and abnormal data whose abnormal type in the data to be processed is an out-of-bounds abnormality is determined according to the out-of-bounds threshold.

[0016] In one embodiment, the step of determining a missing processing solution corresponding to abnormal data according to field characteristics includes:

[0017] The abnormal data is classified and processed by the first classification model to obtain the missing processing solution corresponding to the abnormal data; the first classification model is a machine learning model that uses the missing processing solution set as the target variable and the field feature as the input data;

[0018] Among them, when the field type is a numerical field, the missing processing solution set includes linear regression processing, nearest neighbor processing and preset filling processing; when the field type is a text field, the missing processing solution set includes logistic regression processing and majority filling processing; when the field type is a time field, the missing processing solution set includes sequence filling processing.

[0019] In one embodiment, the step of determining an out-of-bounds processing solution corresponding to abnormal data according to field characteristics includes:

[0020] The abnormal data is classified and processed through the second classification model to obtain the out-of-bounds processing solution corresponding to the abnormal data; the second classification model is a machine learning model with the out-of-bounds processing solution set as the target variable and the field feature as the input data; the out-of-bounds processing solution set includes tail-shrinking processing.

[0021] In one embodiment, the training process of the machine learning model includes:

[0022] Obtain the field features of the sample data and construct training samples based on the field features;

[0023] Iteratively train the basic machine learning model using training samples to obtain evaluation indicators of the machine learning model; the evaluation indicators include precision and recall rate;

[0024] When the evaluation index is within the preset range, the model parameters of the basic machine learning model are adjusted, and the step of iteratively training the basic machine learning model using the training samples is returned and continued until the evaluation index is within the preset range.

[0025] In a second aspect, the present application also provides a power multi-source data processing device, comprising:

[0026] A feature acquisition module is used to acquire field features of the data to be processed; the field features include the abnormal type of abnormal data in the data to be processed;

[0027] A first determination module is used to determine a missing processing solution corresponding to the abnormal data according to the field characteristics when the abnormal type is a missing abnormality;

[0028] The second determination module is used to determine the cross-border processing scheme corresponding to the abnormal data according to the field characteristics when the abnormal type is an cross-border abnormality;

[0029] The exception handling module is used to process abnormal data through corresponding processing solutions.

[0030] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the method steps in the first aspect when executing the computer program.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements any one of the method steps in the first aspect when the computer program is executed by a processor.

[0032] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any one of the method steps in the first aspect when executed by a processor.

[0033] The above-mentioned power multi-source data processing method, device, computer equipment, computer-readable storage medium and computer program product obtain the abnormal type of abnormal data in the data to be processed. When the abnormal type is a missing abnormality, the missing processing scheme corresponding to the abnormal data is determined according to the field characteristics. When the abnormal type is an out-of-bounds abnormality, the out-of-bounds processing scheme corresponding to the abnormal data is determined according to the field characteristics. The abnormal data is processed through the corresponding processing scheme. The processing method can be adaptively determined for abnormal data of different abnormal types, thereby improving the intelligent processing capability of multi-source data. In addition, the determined processing method can realize automatic processing of abnormal data, thereby improving the flexibility of data preprocessing of multi-source data, so that the multi-source data can be accurately and effectively preprocessed. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0035] FIG1 is a schematic flow chart of a method for processing electric power multi-source data in one embodiment;

[0036] FIG2 is a schematic diagram of a confusion matrix of a binary classification model in one embodiment;

[0037] FIG3 is a schematic diagram of a confusion matrix of a multi-classification model in one embodiment;

[0038] FIG4 is a schematic diagram of a flow chart of a method for processing power multi-source data in one embodiment;

[0039] FIG5 is a structural block diagram of a power multi-source data processing device in one embodiment;

[0040] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] In an exemplary embodiment, as shown in FIG1 , a method for processing multi-source power data is provided, the method comprising the following steps 102 to 108. Among them:

[0043] S102: Acquire field characteristics of the data to be processed; the field characteristics include the abnormal type of abnormal data in the data to be processed.

[0044] Optionally, the data to be processed is multi-source data based on the power planning domain. By identifying the field features of the data to be processed and grouping the abnormal data in the data to be processed based on the field features, the corresponding data cleaning method can be automatically recommended to eliminate the impact of the abnormal data. When obtaining the field features, it can be determined based on the field name actively input by the user, or it can be determined by matching with the database.

[0045] Optionally, the field feature includes anomaly types of abnormal data in the data to be processed, wherein the anomaly types are divided into missing anomalies and out-of-bounds anomalies. Missing anomalies refer to the existence of blank fields in the data to be processed, and out-of-bounds anomalies refer to the existence of data points with obvious deviations compared to other data points.

[0046] S104: When the exception type is a missing exception, determine a missing processing solution corresponding to the abnormal data according to the field characteristics.

[0047] Optionally, for abnormal data whose exception type is missing exception, the corresponding missing processing solution usually includes filling the missing data based on other data. In actual applications, the field features are processed by a classification model that matches the missing exception, and the output of the model is a recommended missing processing solution. The missing processing solution output by the model may include multiple missing processing solutions, and the user confirms which missing processing solution to use for processing.

[0048] S106: When the exception type is an out-of-bounds exception, determine an out-of-bounds processing solution corresponding to the abnormal data according to the field characteristics.

[0049] Optionally, for abnormal data whose exception type is an out-of-bounds exception, the corresponding out-of-bounds processing solution usually includes replacing the out-of-bounds data based on other data to reduce the impact of the out-of-bounds value on the analysis results. In actual applications, the field features are processed by a classification model that matches the out-of-bounds exception, and the output of the model is a recommended out-of-bounds processing solution, where the out-of-bounds processing solution output by the model may include multiple ones, and the user confirms which out-of-bounds processing solution to adopt for processing.

[0050] S108: Process the abnormal data through a corresponding processing solution.

[0051] Optionally, after the final processing scheme is determined, the abnormal data is processed based on the processing scheme, which can effectively eliminate the influence of the abnormal data on the analysis results and achieve effective preprocessing of the data to be processed.

[0052] In the above-mentioned power multi-source data processing method, by obtaining the abnormal type of abnormal data in the data to be processed, when the abnormal type is a missing abnormality, the missing processing scheme corresponding to the abnormal data is determined according to the field characteristics; when the abnormal type is an out-of-bounds abnormality, the out-of-bounds processing scheme corresponding to the abnormal data is determined according to the field characteristics. The abnormal data is processed through the corresponding processing scheme, and the processing method can be adaptively determined for abnormal data of different abnormal types, thereby improving the intelligent processing capability of multi-source data. In addition, the determined processing method can realize automatic processing of abnormal data, thereby improving the flexibility of data preprocessing of multi-source data, so that the multi-source data can be accurately and effectively preprocessed.

[0053] In an exemplary embodiment, the field characteristics also include the field type of the data to be processed; the step of obtaining the field characteristics of the data to be processed includes: identifying abnormal data and abnormal types of abnormal data in the data to be processed; performing field statistics on the data to be processed to obtain the field type of the data to be processed.

[0054] Optionally, the field type is used to characterize the field type of the data to be processed. Taking the multi-source power data as an example, the corresponding field types can generally include numerical values, texts, and times, etc. By performing field statistics on the data to be processed, the field type corresponding to each data point can be obtained.

[0055] Optionally, the abnormal data types generally include missing anomalies and out-of-bounds anomalies. In some embodiments, the abnormal data with missing anomalies can be directly counted from the field of the data to be processed, and the abnormal data with out-of-bounds anomalies can generally be determined by box plots, standard deviation method, Mahalanobis distance, and standard score (Z-Score).

[0056] Among them, the box plot is a visualization method for identifying out-of-bounds values. It shows the distribution of data by drawing a five-number summary of the data (minimum value, first quartile, median, third quartile and maximum value). The "abnormal" in the box plot indicates out-of-bounds values, that is, data points that exceed the upper and lower boundaries. Standard deviation is an indicator of the degree of data dispersion. By calculating the standard deviation between a data point and the mean, it can be determined whether the data point is an out-of-bounds value. Generally speaking, data points that are more than n times the standard deviation from the mean can be considered out-of-bounds values, where n is a threshold, usually 3 or greater. Mahalanobis distance is a distance measurement method in multivariate space that can take into account the correlation between various variables. By calculating the Mahalanobis distance between a data point and the mean, it can be determined whether a data point is an out-of-bounds value. Similarly, data points that are more than n times the Mahalanobis distance from the mean can be considered out-of-bounds values. Z-Score is a method to measure the degree of deviation of a data point from the mean. By calculating the Z-Score between a data point and the mean, it can be determined whether a data point is an out-of-bounds value. Generally speaking, data points with a Z-Score greater than 3 or less than -3 can be considered out-of-bounds values.

[0057] In this embodiment, by performing field statistics on the data to be processed, the accurate field type can be obtained, and then the out-of-bounds processing solution corresponding to the abnormal data can be determined according to the field characteristics. The processing method can be adaptively determined for abnormal data of different abnormal types, thereby improving the preprocessing efficiency of multi-source data and realizing intelligent processing of multi-source data.

[0058] In an exemplary embodiment, the field features also include an upper quartile and a lower quartile; the step of identifying abnormal data and abnormal types of abnormal data in the data to be processed includes: taking the upper quartile and the lower quartile as boundary values, calculating the target distance between each data point in the data to be processed and the boundary value; determining an out-of-bounds threshold based on the target distance, and determining, based on the out-of-bounds threshold, abnormal data whose abnormal type in the data to be processed is an out-of-bounds abnormality.

[0059] Optionally, when the acquired field features also include the upper quartile and the lower quartile, the out-of-bounds abnormal data can also be determined by the interquartile range (IQR) method, wherein the upper quartile (i.e., the third quartile) means that 75% of the data points in the data to be processed are less than or equal to this value, and the lower quartile (i.e., the first quartile) means that 25% of the data points in the data to be processed are less than or equal to this value.

[0060] Furthermore, the IQR, that is, the difference between the third quartile (Q3) and the first quartile (Q1), is first calculated, that is, IQR = Q3 - Q1. IQR represents the dispersion of the middle 50% of the data set. After that, Q1 and Q3 are usually expanded by 1.5 times the IQR to define the "normal" range in the data. Any value less than Q1 - 1.5 * IQR or greater than Q3 + 1.5 * IQR is considered to be out-of-bounds (i.e., abnormal data).

[0061] In this embodiment, the upper quartile and the lower quartile are used as boundary values ​​to calculate the target distance between each data point in the data to be processed and the boundary value, and the out-of-bounds threshold is determined according to the target distance. According to the out-of-bounds threshold, the abnormal data whose abnormal type in the data to be processed is an out-of-bounds abnormality can be accurately determined, thereby reducing the impact of the out-of-bounds value on the analysis results.

[0062] In an exemplary embodiment, the step of determining a missing processing solution corresponding to abnormal data based on field characteristics includes: classifying the abnormal data through a first classification model to obtain a missing processing solution corresponding to the abnormal data; the first classification model is a machine learning model with a missing processing solution set as the target variable and field characteristics as input data; wherein, when the field type is a numerical field, the missing processing solution set includes linear regression processing, nearest neighbor processing and preset filling processing; when the field type is a text field, the missing processing solution set includes logistic regression processing and majority filling processing; when the field type is a time field, the missing processing solution set includes sequence filling processing.

[0063] Optionally, based on the field type, the field features may also include features related to abnormal data, and the field features are used as input data for the first classification model, and the first classification model is used to group missing values ​​and adapt the missing value processing scheme corresponding to the missing values. Since the field types mainly include numerical values, texts, and time, etc., the processing schemes for abnormal data of different field types may be the same or different. The field features are shown in Table 1.

[0064] Table 1 Field feature table

[0065]

[0066] Optionally, for abnormal data whose field type is a numerical field, the missing data processing solution set may include linear regression processing, nearest neighbor processing, preset filling processing, etc., as shown in Table 2.

[0067] Table 2 Table of treatment plans for missing values ​​of numerical values

[0068]

[0069] Optionally, for abnormal data whose field type is a text field, the missing data processing solution set may include logistic regression processing and mode filling processing, as shown in Table 3.

[0070] Table 3 Table of text missing value processing solutions

[0071]

[0072] Optionally, for abnormal data whose field type is a time field, its missing processing solution set may include sequence filling processing, etc., as shown in Table 4.

[0073] Table 4 Table of treatment plans for missing values ​​of time category

[0074]

[0075] In this embodiment, the abnormal data with the abnormal type of missing abnormality is grouped by the first classification model, and the processing method can be adaptively determined, which improves the flexibility of data preprocessing of multi-source data, so that the multi-source data can be accurately and effectively preprocessed.

[0076] In an exemplary embodiment, the step of determining an out-of-bounds processing solution corresponding to abnormal data based on field characteristics includes: classifying the abnormal data through a second classification model to obtain an out-of-bounds processing solution corresponding to the abnormal data; the second classification model is a machine learning model with an out-of-bounds processing solution set as the target variable and field characteristics as input data; the out-of-bounds processing solution set includes tail-shrinking processing.

[0077] Optionally, the field feature is used as input data of the second classification model, and the second classification model is used to group the out-of-bounds values ​​and adapt the out-of-bounds processing scheme corresponding to the out-of-bounds values, wherein the out-of-bounds values ​​can usually be processed by shrinking the tail to handle extreme values. Specifically, each variable in the data set is divided into two parts according to a certain percentage (usually 5%): values ​​less than or equal to the 1st percentile and values ​​greater than or equal to the 99th percentile, and the values ​​on the boundary of these two parts are replaced with the average of all values ​​in the part. For example, if the 10th minimum value and the 10th maximum value of a variable are replaced by the average of the first 9 minimum values ​​and the first 9 maximum values ​​of the variable, respectively, then the 10th minimum value of the variable is replaced by a new value, thereby reducing the impact of the out-of-bounds value on the analysis results. It should be noted that since shrinking may cause data distortion, the row where the out-of-bounds value is located can be directly deleted when processing out-of-bounds data.

[0078] In this embodiment, the abnormal data with the exception type of out-of-bounds exception is grouped by the second classification model, and the processing method can be adaptively determined, which improves the flexibility of data preprocessing of multi-source data, so that the multi-source data can be accurately and effectively preprocessed.

[0079] In an exemplary embodiment, the training process of the machine learning model includes: obtaining field features of sample data and constructing training samples based on the field features; iteratively training the basic machine learning model using the training samples to obtain evaluation indicators of the machine learning model; the evaluation indicators include precision and recall; when the evaluation indicators are within a preset range, adjusting the model parameters of the basic machine learning model, returning to the step of iteratively training the basic machine learning model using the training samples, and continuing to execute until the evaluation indicators are within the preset range.

[0080] Optionally, the evaluation indicators for machine learning models usually include accuracy, precision, recall, and F1-score. Accuracy is a common indicator in classification problems, which is used to calculate the ratio of the number of correctly classified predictions to the total number of predictions. For unbalanced data sets, it is also necessary to combine Precision, Recall, and F1-score for evaluation. F1-score is the harmonic mean of Precision and Recall.

[0081] Optionally, in a binary classification problem, assuming that there are two categories of samples: positive samples (Positive) and negative samples (Negative). When the classifier prediction is completed, a confusion matrix can be drawn as shown in Figure 2, where the classification results include the following: True Positive (TP): positive samples are successfully predicted as positive; True Negative (TN): negative samples are successfully predicted as negative; False Positive (FP): negative samples are incorrectly predicted as positive; False Negative (FN): positive samples are incorrectly predicted as negative.

[0082] It should be noted that the data in Figure 2 is only for illustrative purposes. In the binary classification model, the definitions of Accuracy, Precision, Recall, and F1-score are as shown in the following formula:

[0083]

[0084] Among them, Precision focuses on evaluating the proportion of true Positive sample data among all sample data predicted as Positive, while Recall focuses on evaluating how much data is successfully predicted as Positive among all Positive data.

[0085] Among them, when the cost of False Negative (FN) is very high and you hope to avoid FN as much as possible, you should focus on improving the Recall indicator. When the cost of False Positive (FP) is very high, that is, you hope to avoid FP as much as possible, you should focus on improving the Precision indicator. The F1-score is a combination of Precision and Recall.

[0086] Based on this, since the machine learning model in the embodiment of the present application is a multi-classification model, when obtaining evaluation indicators, it is necessary to pay attention to the evaluation indicators of the multi-classification model, and its confusion matrix is ​​shown in Figure 3. The correct classification samples (real samples = predicted samples) in Figure 3 are distributed on the diagonal from the upper left to the lower right.

[0087] Among them, Accuracy is defined as the ratio of the number of samples classified correctly (on the diagonal) to the total number of samples. Accuracy measures the global sample prediction situation, while for Precision and Recall, each category needs to calculate its Precision and Recall separately.

[0088] Furthermore, the comprehensive prediction performance of the machine learning model can be evaluated by methods including but not limited to macro-average method, weighted average method and micro-average method. Among them, the macro-average method refers to directly adding up the evaluation indicators (Precision / Recall / F1-score) of different categories and giving all categories the same weight. The weighted average method refers to determining the corresponding weights according to the actual distribution ratio of each category for weighted averaging. The micro-average method refers to first adding the TP, FP, and FN of each category, and then calculating them according to the formula for binary classification.

[0089] It should be noted that the specific model evaluation method can be selected according to the actual needs of the selected machine learning model, and is not specifically limited here. For example, a random forest classification model can be selected as the prediction model of the preprocessing model, and the preprocessing model is predicted and recommended for multi-source data of different categories including missing values, out-of-bounds values, etc. In the model pre-training stage, the macro-average method is used to evaluate the model to determine whether the prediction performance of the model meets the requirements.

[0090] In this embodiment, by evaluating the predictive ability of the machine learning model based on precision and recall, the accuracy of the model can be improved, so that multi-source data can be accurately and effectively preprocessed.

[0091] In an exemplary embodiment, as shown in FIG4 , a method for processing power multi-source data is provided, the method comprising the following steps:

[0092] Obtain field features of sample data, and construct training samples based on the field features; use the training samples to iteratively train the basic machine learning model to obtain evaluation indicators of the machine learning model; the evaluation indicators include precision and recall; when the evaluation indicators are within the preset range, adjust the model parameters of the basic machine learning model, return to the step of iteratively training the basic machine learning model using the training samples, and continue to execute until the evaluation indicators are within the preset range.

[0093] The upper quartile and the lower quartile are used as boundary values, and the target distance between each data point in the data to be processed and the boundary value is calculated; the cross-boundary threshold is determined according to the target distance, and the abnormal data whose abnormal type in the data to be processed is the cross-boundary abnormality is determined according to the cross-boundary threshold.

[0094] Perform field statistics on the data to be processed and obtain the field type of the data to be processed.

[0095] When the anomaly type is a missing anomaly, the abnormal data is classified and processed through the first classification model to obtain the missing processing plan corresponding to the abnormal data; the first classification model is a machine learning model with a missing processing plan set as the target variable and field features as input data; wherein, when the field type is a numerical field, the missing processing plan set includes linear regression processing, nearest neighbor processing and preset filling processing; when the field type is a text field, the missing processing plan set includes logistic regression processing and majority filling processing; when the field type is a time field, the missing processing plan set includes sequence filling processing.

[0096] When the exception type is an out-of-bounds exception, the abnormal data is classified and processed through the second classification model to obtain the out-of-bounds processing solution corresponding to the abnormal data; the second classification model is a machine learning model with the out-of-bounds processing solution set as the target variable and the field features as the input data; the out-of-bounds processing solution set includes tail-shrinking processing.

[0097] The abnormal data is processed through corresponding processing schemes.

[0098] In this embodiment, by obtaining the exception type of the abnormal data in the data to be processed, when the exception type is a missing exception, the missing processing solution corresponding to the abnormal data is determined according to the field characteristics; when the exception type is an out-of-bounds exception, the out-of-bounds processing solution corresponding to the abnormal data is determined according to the field characteristics. The abnormal data is processed through the corresponding processing solution, and the processing method can be adaptively determined for abnormal data of different exception types, thereby improving the intelligent processing capability of multi-source data. In addition, the determined processing method can realize automatic processing of abnormal data, thereby improving the flexibility of data preprocessing of multi-source data, so that the multi-source data can be accurately and effectively preprocessed.

[0099] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0100] Based on the same inventive concept, the embodiment of the present application also provides a power multi-source data processing device for implementing the power multi-source data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more power multi-source data processing device embodiments provided below can refer to the limitations of the power multi-source data processing method above, and will not be repeated here.

[0101] In an exemplary embodiment, as shown in FIG5 , a power multi-source data processing device is provided, comprising: a feature acquisition module 10, a first determination module 20, a second determination module 30 and an exception processing module 40, wherein:

[0102] The feature acquisition module 10 is used to acquire field features of the data to be processed; the field features include the abnormal type of abnormal data in the data to be processed.

[0103] The first determination module 20 is used to determine a missing processing solution corresponding to the abnormal data according to the field characteristics when the abnormal type is a missing abnormality.

[0104] The second determination module 30 is used to determine an out-of-bounds processing solution corresponding to the abnormal data according to the field characteristics when the abnormal type is an out-of-bounds abnormality.

[0105] The exception processing module 40 is used to process the abnormal data through a corresponding processing solution.

[0106] In an exemplary embodiment, the field feature also includes the field type of the data to be processed; the feature acquisition module 10 is also used to identify abnormal data and abnormal types of abnormal data in the data to be processed; and field statistics are performed on the data to be processed to obtain the field type of the data to be processed.

[0107] In an exemplary embodiment, the field features also include an upper quartile and a lower quartile; the feature acquisition module 10 is also used to calculate the target distance between each data point in the data to be processed and the boundary value using the upper quartile and the lower quartile as boundary values; determine the out-of-bounds threshold based on the target distance, and determine the abnormal data in the data to be processed whose abnormal type is an out-of-bounds exception based on the out-of-bounds threshold.

[0108] In an exemplary embodiment, the first determination module 20 is also used to classify the abnormal data through the first classification model to obtain the missing processing solution corresponding to the abnormal data; the first classification model is a machine learning model with a missing processing solution set as the target variable and field features as input data; wherein, when the field type is a numerical field, the missing processing solution set includes linear regression processing, nearest neighbor processing and preset filling processing; when the field type is a text field, the missing processing solution set includes logistic regression processing and majority filling processing; when the field type is a time field, the missing processing solution set includes sequence filling processing.

[0109] In an exemplary embodiment, the second determination module 30 is also used to classify the abnormal data through a second classification model to obtain an out-of-bounds processing solution corresponding to the abnormal data; the second classification model is a machine learning model with an out-of-bounds processing solution set as the target variable and field features as input data; the out-of-bounds processing solution set includes tail-shrinking processing.

[0110] In an exemplary embodiment, the exception handling module 40 is also used to obtain field features of sample data and construct training samples based on the field features; iteratively train the basic machine learning model using the training samples to obtain evaluation indicators of the machine learning model; the evaluation indicators include precision and recall; when the evaluation indicators are within a preset range, adjust the model parameters of the basic machine learning model, return to the step of iteratively training the basic machine learning model using the training samples, and continue to execute until the evaluation indicators are within the preset range.

[0111] Each module in the above-mentioned power multi-source data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0112] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for processing multi-source data of electric power is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0113] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining field characteristics of data to be processed; the field characteristics include the exception type of abnormal data in the data to be processed; when the exception type is a missing exception, determining a missing processing solution corresponding to the abnormal data according to the field characteristics; when the exception type is an out-of-bounds exception, determining an out-of-bounds processing solution corresponding to the abnormal data according to the field characteristics; and processing the abnormal data through the corresponding processing solution.

[0115] In one embodiment, the field characteristics also include the field type of the data to be processed; the field characteristics of the data to be processed involved in obtaining the field characteristics of the data to be processed when the processor executes the computer program include: identifying abnormal data and abnormal types of abnormal data in the data to be processed; performing field statistics on the data to be processed to obtain the field type of the data to be processed.

[0116] In one embodiment, the field features also include an upper quartile and a lower quartile; the identification of abnormal data and abnormal types of abnormal data in the data to be processed involved when the processor executes a computer program includes: taking the upper quartile and the lower quartile as boundary values, calculating the target distance between each data point in the data to be processed and the boundary value; determining an out-of-bounds threshold based on the target distance, and determining, based on the out-of-bounds threshold, abnormal data whose abnormal type in the data to be processed is an out-of-bounds abnormality.

[0117] In one embodiment, when a processor executes a computer program, the method involved in determining a missing processing solution corresponding to abnormal data based on field characteristics includes: classifying the abnormal data through a first classification model to obtain a missing processing solution corresponding to the abnormal data; the first classification model is a machine learning model with a missing processing solution set as a target variable and field characteristics as input data; wherein, when the field type is a numerical field, the missing processing solution set includes linear regression processing, nearest neighbor processing and preset filling processing; when the field type is a text field, the missing processing solution set includes logistic regression processing and mode filling processing; when the field type is a time field, the missing processing solution set includes sequence filling processing.

[0118] In one embodiment, when a processor executes a computer program, the process of determining an out-of-bounds processing solution corresponding to abnormal data based on field characteristics includes: classifying the abnormal data through a second classification model to obtain an out-of-bounds processing solution corresponding to the abnormal data; the second classification model is a machine learning model that uses an out-of-bounds processing solution set as a target variable and field characteristics as input data; the out-of-bounds processing solution set includes shrinking processing.

[0119] In one embodiment, the training process of the machine learning model involved in the execution of a computer program by the processor includes: obtaining field features of sample data, and constructing training samples based on the field features; iteratively training the basic machine learning model using the training samples, and obtaining evaluation indicators of the machine learning model; the evaluation indicators include precision and recall; when the evaluation indicators are within a preset range, adjusting the model parameters of the basic machine learning model, returning to the step of iteratively training the basic machine learning model using the training samples, and continuing to execute until the evaluation indicators are within the preset range.

[0120] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining field characteristics of data to be processed; the field characteristics include the exception type of abnormal data in the data to be processed; when the exception type is a missing exception, determining a missing processing solution corresponding to the abnormal data according to the field characteristics; when the exception type is an out-of-bounds exception, determining an out-of-bounds processing solution corresponding to the abnormal data according to the field characteristics; and processing the abnormal data through the corresponding processing solution.

[0121] In one embodiment, the field characteristics also include the field type of the data to be processed; when the computer program is executed by the processor, obtaining the field characteristics of the data to be processed includes: identifying abnormal data and abnormal types of abnormal data in the data to be processed; performing field statistics on the data to be processed to obtain the field type of the data to be processed.

[0122] In one embodiment, the field features also include an upper quartile and a lower quartile; the computer program executed by the processor involves identifying abnormal data and abnormal types of abnormal data in the data to be processed, including: taking the upper quartile and the lower quartile as boundary values, calculating the target distance between each data point in the data to be processed and the boundary value; determining an out-of-bounds threshold based on the target distance, and determining, based on the out-of-bounds threshold, abnormal data whose abnormal type in the data to be processed is an out-of-bounds abnormality.

[0123] In one embodiment, when a computer program is executed by a processor, the method involved in determining the missing processing solution corresponding to abnormal data based on field characteristics includes: classifying the abnormal data through a first classification model to obtain the missing processing solution corresponding to the abnormal data; the first classification model is a machine learning model with a missing processing solution set as the target variable and field characteristics as input data; wherein, when the field type is a numerical field, the missing processing solution set includes linear regression processing, nearest neighbor processing and preset filling processing; when the field type is a text field, the missing processing solution set includes logistic regression processing and majority filling processing; when the field type is a time field, the missing processing solution set includes sequence filling processing.

[0124] In one embodiment, when a computer program is executed by a processor, the process involved in determining an out-of-bounds processing solution corresponding to abnormal data based on field features includes: classifying the abnormal data through a second classification model to obtain an out-of-bounds processing solution corresponding to the abnormal data; the second classification model is a machine learning model that uses an out-of-bounds processing solution set as a target variable and field features as input data; the out-of-bounds processing solution set includes shrinking processing.

[0125] In one embodiment, the training process of the machine learning model involved when the computer program is executed by the processor includes: obtaining field features of sample data, and constructing training samples based on the field features; iteratively training the basic machine learning model using the training samples to obtain evaluation indicators of the machine learning model; the evaluation indicators include precision and recall; when the evaluation indicators are within a preset range, adjusting the model parameters of the basic machine learning model, returning to the step of iteratively training the basic machine learning model using the training samples, and continuing to execute until the evaluation indicators are within the preset range.

[0126] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: obtaining field characteristics of data to be processed; the field characteristics include the exception type of abnormal data in the data to be processed; when the exception type is a missing exception, determining a missing processing solution corresponding to the abnormal data according to the field characteristics; when the exception type is an out-of-bounds exception, determining an out-of-bounds processing solution corresponding to the abnormal data according to the field characteristics; and processing the abnormal data through the corresponding processing solution.

[0127] In one embodiment, the field characteristics also include the field type of the data to be processed; when the computer program is executed by the processor, obtaining the field characteristics of the data to be processed includes: identifying abnormal data and abnormal types of abnormal data in the data to be processed; performing field statistics on the data to be processed to obtain the field type of the data to be processed.

[0128] In one embodiment, the field features also include an upper quartile and a lower quartile; the computer program executed by the processor involves identifying abnormal data and abnormal types of abnormal data in the data to be processed, including: taking the upper quartile and the lower quartile as boundary values, calculating the target distance between each data point in the data to be processed and the boundary value; determining an out-of-bounds threshold based on the target distance, and determining, based on the out-of-bounds threshold, abnormal data whose abnormal type in the data to be processed is an out-of-bounds abnormality.

[0129] In one embodiment, when a computer program is executed by a processor, the method involved in determining the missing processing solution corresponding to abnormal data based on field characteristics includes: classifying the abnormal data through a first classification model to obtain the missing processing solution corresponding to the abnormal data; the first classification model is a machine learning model with a missing processing solution set as the target variable and field characteristics as input data; wherein, when the field type is a numerical field, the missing processing solution set includes linear regression processing, nearest neighbor processing and preset filling processing; when the field type is a text field, the missing processing solution set includes logistic regression processing and majority filling processing; when the field type is a time field, the missing processing solution set includes sequence filling processing.

[0130] In one embodiment, when a computer program is executed by a processor, the process involved in determining an out-of-bounds processing solution corresponding to abnormal data based on field features includes: classifying the abnormal data through a second classification model to obtain an out-of-bounds processing solution corresponding to the abnormal data; the second classification model is a machine learning model that uses an out-of-bounds processing solution set as a target variable and field features as input data; the out-of-bounds processing solution set includes shrinking processing.

[0131] In one embodiment, the training process of the machine learning model involved when the computer program is executed by the processor includes: obtaining field features of sample data, and constructing training samples based on the field features; iteratively training the basic machine learning model using the training samples to obtain evaluation indicators of the machine learning model; the evaluation indicators include precision and recall; when the evaluation indicators are within a preset range, adjusting the model parameters of the basic machine learning model, returning to the step of iteratively training the basic machine learning model using the training samples, and continuing to execute until the evaluation indicators are within the preset range.

[0132] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing multi-source power data, characterized in that: The method comprises: Acquire field characteristics of the data to be processed; the field characteristics include the abnormal type of abnormal data in the data to be processed; In the case where the exception type is a missing exception, determining a missing processing solution corresponding to the abnormal data according to the field characteristics; In the case where the exception type is an out-of-bounds exception, determining an out-of-bounds processing solution corresponding to the abnormal data according to the field characteristics; The abnormal data is processed through a corresponding processing scheme.

2. The method according to claim 1, characterized in that The field characteristics also include the field type of the data to be processed; the step of obtaining the field characteristics of the data to be processed includes: Identifying abnormal data in the data to be processed and the abnormal type of the abnormal data; Perform field statistics on the data to be processed to obtain the field type of the data to be processed.

3. The method according to claim 2, characterized in that The field feature also includes an upper quartile and a lower quartile; the identifying of abnormal data in the data to be processed and the abnormal type of the abnormal data includes: Taking the upper quartile and the lower quartile as boundary values, calculating the target distance between each data point in the data to be processed and the boundary value; An out-of-bounds threshold is determined according to the target distance, and abnormal data whose abnormal type in the data to be processed is an out-of-bounds abnormality is determined according to the out-of-bounds threshold.

4. The method according to claim 2, characterized in that: The determining, according to the field characteristics, a missing processing scheme corresponding to the abnormal data includes: Classify the abnormal data through a first classification model to obtain a missing processing solution corresponding to the abnormal data; the first classification model is a machine learning model that uses a missing processing solution set as a target variable and the field feature as input data; Among them, when the field type is a numerical field, the missing processing solution set includes linear regression processing, nearest neighbor processing and preset filling processing; when the field type is a text field, the missing processing solution set includes logistic regression processing and mode filling processing; when the field type is a time field, the missing processing solution set includes sequence filling processing.

5. The method according to claim 1, characterized in that The determining, according to the field feature, a cross-border processing scheme corresponding to the abnormal data includes: The abnormal data is classified and processed through a second classification model to obtain an out-of-bounds processing solution corresponding to the abnormal data; the second classification model is a machine learning model with an out-of-bounds processing solution set as the target variable and the field feature as the input data; the out-of-bounds processing solution set includes tail-shrinking processing.

6. The method according to claim 4 or 5, characterized in that: The training process of the machine learning model includes: Acquire field features of sample data, and construct training samples based on the field features; Iteratively training the basic machine learning model using the training samples to obtain evaluation indicators of the machine learning model; the evaluation indicators include precision and recall; When the evaluation index is within the preset range, the model parameters of the basic machine learning model are adjusted, and the step of iteratively training the basic machine learning model using the training samples is returned to and continued until the evaluation index is within the preset range.

7. A power multi-source data processing device, characterized in that: The device comprises: A feature acquisition module, used to acquire field features of the data to be processed; the field features include the abnormal type of abnormal data in the data to be processed; A first determination module, configured to determine, when the exception type is a missing exception, a missing processing solution corresponding to the abnormal data according to the field feature; A second determination module is used to determine, when the exception type is an out-of-bounds exception, an out-of-bounds processing solution corresponding to the abnormal data according to the field characteristics; The exception processing module is used to process the abnormal data through a corresponding processing solution.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.