A method and system for actively auditing abnormal business data of power marketing

Abstract

This invention discloses a proactive auditing method and system for abnormal business data in electricity marketing. The method includes: calculating the information gain ratio among various classification attributes in the given auditable data and analyzing the business correlation of the classification attributes; based on the correlation analysis results among the classification attributes, concatenating attributes with significant correlations to form new mixed data; based on the new mixed data, performing frequency feature transformation on the classification attributes to generate attribute feature data that can be directly input into model training; based on the attribute feature data, constructing and training an enhanced isolated forest model using the enhanced isolated forest algorithm; based on the trained enhanced isolated forest model, calculating the abnormal score of the electricity marketing business data; based on the abnormal score of the electricity marketing business data, adaptively identifying abnormal groups to obtain adaptive abnormal results; and combining the output adaptive abnormal results with the electricity marketing business data to output the final abnormal identification result.

Description

Technical Field

[0001] The present invention belongs to the technical field of power systems, and in particular relates to a method and system for actively auditing abnormal business data of power marketing. Background Art

[0002] In the power marketing business, marketing audit is an important way to improve internal control constraints, discover marketing errors, and strengthen marketing business risk management. With the advancement of information construction in the power sector, it is very necessary to conduct online regular audits of business data such as changes in electricity consumption, business expansion and registration, and electricity price and fee management based on automated means. The information construction of power marketing has entered the fast lane and is currently in a critical period of digital transformation.

[0003] In the previous abnormality auditing methods, business experts often "manually define rules" to check related abnormalities, and situations that do not conform to relevant rules are abnormalities. The existing technical solutions can be generally summarized into the following categories:

[0004] The first category is the method of manually defining rules: In the existing power marketing audit business, the rule-based method is mainly used to audit power business data, for example: anomalies in meter reading and collection: "Anomalies in electricity usage category - the electricity usage category in the customer file information is considered an anomaly if it does not match the electricity price code"; capacity anomalies in user file category: "The power supply capacity in the customer file information is 0 or empty, which is considered an anomaly"; anomalies are checked through rules such as those defined by experts above, and those that do not match the rules are considered an anomaly.

[0005] Disadvantages of manual experience rules: As mentioned above, there are the following main defects in conducting audits by manually defining rules: the definition of rules is done by experts manually, which is a huge workload and requires a lot of manpower and time costs; manpower is finite, and business experts face huge amounts of data and complex anomaly types. It is impossible to formulate complete rules that apply to all abnormal situations, resulting in limited audit scope; rule-based audit methods can only detect anomalies within the rules, such as the capacity anomaly rule for user profiles: "The power supply capacity of 0 or empty in the customer profile information is considered an anomaly." Based on this rule, it can only be detected that the power supply capacity is 0 or empty as an anomaly, but it cannot detect obvious extreme value anomalies, such as 99999, resulting in a low detection rate of abnormal data during audits.

[0006] The second category is supervised machine learning methods: Since the goal of power marketing audit is to identify problematic data, many decision models such as decision trees, random forests, logistic regression and other supervised learning methods have been introduced into the audit field to identify problematic data. Because the audit business scope involves different fields such as business expansion registration, meter reading and collection, customer service and operation, and different fields have different requirements, different models need to be built.

[0007] Disadvantages of supervised methods: The main disadvantages of supervised methods such as logistic regression, decision tree, random forest, etc. are: a corresponding model needs to be built for each target, resulting in poor applicability of the model and inability to adapt to the needs of various business categories faced by the continuous expansion of audit business; supervised methods require that the training data must be labeled when training the model, while the business data generated by the power marketing business is unlabeled and the data volume is huge. If manual labeling is performed, the workload will be huge.

[0008] The third category is unsupervised machine learning methods: Unsupervised learning methods are also widely used in the audit field, mainly using methods such as anomaly detection, such as cluster analysis, LOF (Local outlier factor), isolation forest algorithm, etc.

[0009] Disadvantages of unsupervised methods: Most of the existing unsupervised anomaly detection methods are only applicable to single-type data, but are powerless against mixed data such as power marketing business data; and methods such as LOF and kNN (K-NearestNeighbor) have large computational requirements and cannot meet the high-performance processing requirements of massive data in auditing business; in the study of anomaly detection methods, existing methods such as Isolation Forest (iForest) and POD (Pattern-Based Outlier Detection) can only give anomaly scores, but cannot specifically identify whether each piece of data is abnormal. In addition, the classic iForest algorithm can only process numerical data and cannot meet the processing requirements of rich business data.

[0010] In the power marketing business, marketing audit work is an important way to improve internal control constraints, discover marketing errors, and strengthen marketing business risk management. The business data of power marketing includes customer file data for business expansion and registration management, and meter reading and fee calculation data for electricity price and electricity fee management. These data are structured data stored in relational databases. In the process of entering relevant data into the database, incorrect data may be input due to operational errors of business personnel, insufficient understanding of relevant business policies, or data collection and transmission failures. With the advancement of informatization construction in the power sector, it is very necessary to use informatization and automation means to conduct online regular audits on business data such as changes in electricity consumption, business expansion and registration, electricity prices and electricity fee management, and identify problematic data to improve the compliance of marketing business.

[0011] In summary, the current power marketing audit business urgently needs a heuristic audit method in addition to rule-driven auditing to proactively identify anomalies for which rules have not yet been clearly defined. Summary of the invention

[0012] In order to overcome the problems in the prior art, the present invention proposes a method and system for actively auditing abnormal business data of power marketing.

[0013] The technical solution of the present invention to solve the above technical problems is as follows:

[0014] A method for actively auditing abnormal business data of power marketing, comprising the following steps:

[0015] Step 1. For the given power marketing business data to be audited, the information gain rate is used as an evaluation index to calculate the correlation between classification attributes;

[0016] Step 2. Based on the correlation analysis results between the classification attributes in the power marketing business data, the classification attributes with significant correlation are spliced ​​to generate new classification attributes, and the original classification attributes are deleted to form new mixed data;

[0017] Step 3. Based on the new mixed data, perform frequency feature transformation on the categorical attributes, convert the categorical data into numerical data, and generate attribute feature data that can be directly input into model training;

[0018] Step 4. Based on the attribute feature data, the enhanced isolation forest model is constructed and trained based on the enhanced isolation forest algorithm;

[0019] Step 5. Based on the trained enhanced isolation forest model, calculate the anomaly score of the power marketing business data;

[0020] Step 6. Based on the abnormal scores of power marketing business data, a unary beta mixed model is constructed to perform adaptive abnormal group discrimination;

[0021] Step 7. Pass ICL_BIC The Integrated Completed Likelihood - Bayesian Information Criterion identifies the best univariate beta mixture model and uses its output as the final output of the anomaly result.

[0022] Furthermore, the step 1 uses the information gain rate as an evaluation index to calculate the correlation between classification attributes for the given power marketing business data to be audited, specifically including:

[0023] Step 1-1. Calculate the information gain rate between various classification attributes in the power marketing business data to be audited. The calculation formula of the information gain rate is:

[0024]

[0025]

[0026]

[0027] in, and Represent two different classification attributes, , Respectively represent the information gain rate and information gain between the classification attributes. Represents a calculated property right The information gain rate of and They represent the calculation of information entropy and conditional entropy respectively. Indicates that the calculation is in the attribute Under the condition, the property Information entropy of Represents a categorical attribute The m attribute values, Indicated in the attribute In the calculation, it is equal to The ratio of the number of attribute values ​​to the total attribute length;

[0028] Step 1-2. Convert the information gain rate between the classification attributes into the correlation between the classification attributes. The formula is as follows:

[0029]

[0030] in, The value range is [0,1], indicating a classification attribute and The correlation between Larger values ​​indicate greater correlation.

[0031] Furthermore, in step 4, based on the attribute feature data, an enhanced isolation forest model is constructed and trained based on the enhanced isolation forest algorithm. The modeling process is as follows:

[0032] Step 4-1. Set the sampling number or proportion and the number of isolated trees required to build the model;

[0033] Step 4-2. Sampling: Randomly extract a set number or proportion of sub-samples from the power marketing business data; for each segmentation tree, set a root node and use the root node as the current node;

[0034] Step 4-3. Attribute segmentation: On the current node, randomly select several attributes as target attributes for segmentation, and construct the left subtree and right subtree of the segmentation node respectively according to the segmentation results;

[0035] The segmentation strategy is as follows:

[0036] 0

[0037] in, Represents a vector consisting of several target attributes; represents the intercept vector, which is obtained by taking values ​​from a uniform distribution between the maximum and minimum values ​​of each target attribute; The normal vector, i.e. the slope of the splitting plane, is composed by randomly generating a value for each target attribute from the standard normal distribution N(0,1);

[0038] Then by calculating Get the splitting hyperplane, and split the sample set of the current node based on the splitting hyperplane. If the splitting strategy of the target attribute value of the current node is less than 0, that is, it is below the splitting hyperplane, then the samples corresponding to the target attribute fall into the left subtree, and the rest fall into the right subtree;

[0039] Step 4-4. Build an isolated tree; repeat step 4-3 in the child node until there is only one data in the child node or it is impossible to continue cutting or the child node has reached the maximum depth of the set tree, then stop the step and complete the construction of the isolated tree;

[0040] Step 4-5. Construct an enhanced isolation forest; according to the sampling number / ratio set in step 4-1 and the number of trees to be constructed, loop steps 4-2, 4-3, and 4-4 to complete the construction of all isolated trees to form an enhanced isolation forest.

[0041] Furthermore, in step 5, based on the trained enhanced isolation forest model, the anomaly score of the power marketing business data is calculated, which specifically includes the following sub-steps:

[0042] Step 5-1. Input all the data to be audited into the enhanced isolation forest model. For each data point, after passing through the enhanced isolation forest model composed of T trees, T segmentation results will be obtained;

[0043] Step 5-2: For each data point, calculate the path length of the data point in each isolated tree. h(x) , that is, the number of edges that the data point passes through from the root node of the tree to the leaf node;

[0044] Step 5-3. Calculate the average path length of each data point in the enhanced isolation forest, that is, in all trees. The calculation formula is:

[0045]

[0046] in, E(h(x)) represents the average path length, T represents the number of trees in the enhanced isolation forest, h t (x) Indicates that the data point is t The path length on the tree;

[0047] Step 5-4. Normalize the average path length of all data points and calculate the anomaly score of each data point. The calculation formula is:

[0048]

[0049] in, n Indicates the amount of data for the input sample when building the tree. c(n) is the global average path length, which is used for normalization processing. Its formula is:

[0050]

[0051] in, , is Euler's constant.

[0052] Furthermore, in step 6, based on the abnormality score of the power marketing business data, a univariate beta mixture model is constructed to perform adaptive abnormal group discrimination, specifically including:

[0053] Step 6-1. Use the k-means method to perform cluster analysis on the abnormal score results, and divide the abnormal scores of the power marketing business data to be audited into k components;

[0054] Step 6-2. Construct a univariate beta mixture model and output the univariate beta mixture model and abnormal results.

[0055] Furthermore, the univariate beta mixed model includes:

[0056] Univariate beta mixture distribution:

[0057]

[0058] in, It is m A univariate beta distribution with components, V i represents the anomaly score vector, and represents the parameters of the univariate beta distribution with M components; represents the mixing coefficient between the M components, and ;

[0059] The probability density function of the univariate beta distribution is:

[0060]

[0061] in, represents the gamma function;

[0062] Using maximum likelihood estimation to estimate the parameters of the univariate beta mixed model, the log-likelihood function is:

[0063]

[0064]

[0065] in, represents taking the logarithm, V represents the anomaly score vector, parameter set represents the set of parameters that need to be estimated; and N Anomaly score vector V Associated, used to indicate anomaly scores V i belonging, Indicates i Anomaly score V i Does it belong to m components, if =1, it means it belongs to, =0 does not apply.

[0066] Furthermore, in step 7, the ICL-BIC criterion is used to find the optimal model, and its calculation formula is:

[0067]

[0068] in, Represents the parameter set The estimate, Represents the belonging vector The estimate, represents the number of parameters of a model with M components, Indicates the amount of data.

[0069] The present invention also discloses an active audit system for abnormal business data of power marketing, including: a data source management module, a data reading module, a feature processing module, a model training module, an abnormal output module and an abnormal output management module;

[0070] The data source management module is used to manage the database and access information of the data required by the power marketing business data abnormality audit system, and the database table where the marketing business data to be audited is located;

[0071] The data reading module is used to obtain the required data from the data source management module, and perform preprocessing operations including missing value processing, deduplication, and format conversion, and save the processed data uniformly as table data required by the system;

[0072] The feature processing module is used to obtain table data from the data reading module, transform and integrate the table data, calculate the information gain rate of the classification attribute features in the power marketing business data features, evaluate the correlation between the classification attributes and complete the field splicing, and then transform the classification data into numerical data based on the classification attribute frequency to generate attribute feature data that can be directly input into model training;

[0073] The model training module is used to randomly extract a preset number or proportion of data from the data output by the feature processing module for training the enhanced isolation forest algorithm model;

[0074] The abnormal output module is used to output normal values ​​and abnormal values ​​in the power marketing business data;

[0075] The abnormal output management module is used to combine the output abnormal results with the power marketing business data output by the data source management module, and manage the final result presentation and output work.

[0076] Furthermore, the abnormal output module includes:

[0077] Algorithm prediction submodule: It is used to obtain all the data output from the feature processing module based on the model trained by the model training module, input the model for prediction, and output the abnormality score of the power marketing business data;

[0078] Adaptive abnormal group discrimination submodule: Based on the vector of abnormal scores of power marketing business data, for each k Value, proceed k-means Initialize and construct a singleton beta Hybrid models are used to adaptively determine normal and outliers in the data;

[0079] The abnormal result recognition submodule is used to identify multiple unary results based on the output of the adaptive abnormal group discrimination module. beta Mixed models and unusual results, through ICL_BIC The criterion identifies the best hybrid model and uses its output anomaly results as the final output anomaly results.

[0080] Compared with the prior art, the present invention has the following technical effects:

[0081] (1) In order to solve the problem that in the process of auditing power marketing business data, it takes a lot of effort for experts to formulate rules, the summarized rules do not cover all aspects, and the thresholds for abnormal assessment need to be manually defined, which makes it impossible to adaptively identify abnormal data, we propose an unsupervised marketing data audit method that does not require empirical rule-driven and supports different data types. It does not require business experts to formulate prior rules based on their business experience, but uses machine learning methods to actively heuristically screen for abnormalities.

[0082] (2) In view of the interdependence between the classified data in the power marketing business data, and the problem that the existing unsupervised anomaly detection method only supports independent attributes but ignores the correlation between businesses, the present invention proposes a correlation analysis and attribute processing method based on information gain rate, which can better meet the needs of power marketing audit business;

[0083] (3) In view of the problem that the classic isolation forest algorithm cannot process categorical data, the present invention proposes an improved method to transform the frequency characteristics of categorical attributes and convert the categorical attributes into numerical data, so that the algorithm can be applied to mixed data;

[0084] (4) To address the problem that previous unsupervised anomaly detection methods only provide an assessment of the degree of anomaly but cannot identify abnormal individuals, this paper proposes an abnormal result identification strategy based on k-means and univariate beta distribution, which can directly identify abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A schematic diagram of a process flow of a method for proactively auditing abnormal business data of power marketing according to the present invention;

[0086] Figure 2 A block diagram of an active audit system for abnormal business data of power marketing according to the present invention;

[0087] Figure 3 Schematic diagram of the three-dimensional segmentation hyperplane of the present invention. DETAILED DESCRIPTION

[0088] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0089] The present invention focuses on solving the problem that the current power marketing audit business relies too much on manually set rules to check business data, thereby failing to actively discover problematic data outside the rule definition, leading to business risks.

[0090] In view of the above problems, the present invention provides a method for actively auditing abnormal business data of power marketing, the method comprising the following steps:

[0091] Step 1. For the given power marketing business data to be audited, the information gain rate is used as an evaluation index to calculate the correlation between classification attributes;

[0092] Obtain the power marketing business data to be audited, perform preprocessing operations including missing value processing, deduplication, format conversion, etc., and save the processed data as the required table data.

[0093] Due to specific business reasons, there are dependencies between many types of data values ​​in power marketing business data, and this relationship has always been based on business experience. If each attribute is considered independent, the effect of anomaly detection will be greatly reduced, which is also inconsistent with business expectations.

[0094] The present invention first solves the problem of analyzing the potential correlation between the classified attribute services, that is, using the information gain rate as an evaluation index to calculate the correlation between the classified attributes, which specifically includes the following steps:

[0095] Step 1-1. Calculate the information gain rate between various classification attributes in the given power marketing business data to be audited;

[0096] make for N A collection of mixed attribute data points, where A c and A n Represents the set of categorical attributes and continuous attributes respectively.

[0097] For each classification attribute, the information gain rate between it and other classification attributes is calculated, and the correlation between classification attributes is measured based on the information gain rate.

[0098] The calculation formula of information gain rate is:

[0099]

[0100]

[0101]

[0102] in, and Represent two different classification attributes, , Respectively represent the information gain rate and information gain between the classification attributes. Represents a calculated property right The information gain rate of and Respectively represent the calculation of information entropy and conditional entropy, such as This means that the calculation is based on the attribute Under the condition, the property Information entropy of Represents categorical attributes The m attribute values, Indicated in the attribute In the calculation, it is equal to The ratio of the number of attribute values ​​to the total attribute length.

[0103] Step 1-2. Convert the information gain rate between the classification attributes into the correlation between the classification attributes;

[0104] Since the information gain rate is asymmetric, that is, and ( Therefore, it cannot be used directly to measure the correlation between classification attributes and needs further transformation, as shown in the following formula:

[0105]

[0106] in, The value range is [0,1], indicating a classification attribute and The correlation between Larger values ​​indicate greater correlation.

[0107] Step 2. Based on the correlation analysis results between the classification attributes in the power marketing business data, the classification attributes with significant correlation are spliced ​​to generate new classification attributes, and the original classification attributes are deleted to form new mixed data.

[0108] Since there may be associations between classification attributes, especially in power marketing business data, there may be significant correlations between classification attributes, such as strong correspondences between classification attributes "YHLBDM" (user category code) and "JLFSDM" (metering method code), "DJDM" (electricity price code) and "YDLBDM" (electricity category code), etc., in order to be able to retrieve anomalies existing in multidimensional mixtures, the present invention is based on the correlation between the classification attributes calculated in step 1, splices the classification attributes with significant correlation to generate new classification attributes, deletes the original classification attributes, and merges them with continuous attributes. Usually, the correlation threshold can be set to 0.5, and a correlation greater than 0.5 indicates significant correlation, otherwise it is irrelevant / low correlation.

[0109] Step 3. Based on the new mixed data, perform frequency feature transformation on the categorical attributes, convert the categorical data into numerical data, and generate attribute feature data that can be directly input into model training.

[0110] Unlike the classic isolation forest algorithm, which is only applicable to numerical data, in order to make the enhanced isolation forest algorithm in the present invention applicable not only to numerical data but also to categorical data and mixed data of both categorical data and numerical data, the present invention performs frequency feature transformation on categorical data and converts categorical data into numerical data.

[0111] Step 4. Based on the attribute feature data, the enhanced isolation forest model is constructed and trained based on the enhanced isolation forest algorithm.

[0112] The present invention calculates the abnormal score of mixed data based on the enhanced isolation forest algorithm. Since the classic isolation forest algorithm is only applicable to numerical data, the present invention extends the algorithm to support classification attributes.

[0113] The enhanced isolation forest model is constructed and trained based on the enhanced isolation forest algorithm. The modeling process is as follows:

[0114] Step 4-1. Set the sampling number or proportion and the number of isolated trees required to build the model;

[0115] Step 4-2. Sampling: Randomly extract a set number or proportion of sub-samples from the power marketing business data; for each partition tree, set a root node and use the root node as the current node.

[0116] Step 4-3. Attribute segmentation: On the current node, randomly select several attributes as target attributes for segmentation, and construct the left subtree and right subtree of the segmentation node respectively according to the segmentation results.

[0117] The segmentation strategy is as follows:

[0118] 0

[0119] in, Represents a vector consisting of several target attributes; represents the intercept vector, which is obtained by taking values ​​from a uniform distribution between the maximum and minimum values ​​of each target attribute; The normal vector is simply the slope of the splitting plane, which is composed of randomly generating a value for each target attribute from the standard normal distribution N(0,1);

[0120] Then by calculating Get the splitting hyperplane (take the selection of 3 target attributes as an example, the splitting hyperplane is as follows Figure 3As shown). The sample set of the current node is split based on the splitting hyperplane. If the splitting strategy of the target attribute value of the current node is less than 0, that is, it is below the splitting hyperplane, the samples corresponding to the target attribute fall into the left subtree, and the rest fall into the right subtree;

[0121] Step 4-4. Build an isolated tree; repeat step 4-3 in the child node until there is only one data in the child node (cannot continue cutting) or the child node has reached the maximum depth of the set tree, then stop the step and complete the construction of the isolated tree.

[0122] Step 4-5. Construct an enhanced isolation forest; according to the sampling number (or ratio) set in step 4-1 and the number of trees to be constructed, loop steps 4-2, 4-3, and 4-4 to complete the construction of all isolated trees to form an enhanced isolation forest.

[0123] Step 5. Based on the trained enhanced isolation forest model, calculate the anomaly score of the power marketing business data. Step 5 specifically includes the following sub-steps:

[0124] Step 5-1. Input the data to be audited into the enhanced isolation forest model. For each data point, after passing through the enhanced isolation forest model composed of T trees, T segmentation results will be obtained.

[0125] Step 5-2: For each data point, calculate the path length of the data point in each enhanced isolation tree. h(x) , that is, the number of edges that the data point passes through from the root node of the tree to the leaf node.

[0126] Step 5-3. Calculate the average path length of each data point in the enhanced isolation forest (i.e. all trees), the calculation formula is:

[0127]

[0128] in, E(h(x)) represents the average path length, T represents the number of trees in the enhanced isolation forest, h t (x) Indicates that the data point is t The path length on the tree;

[0129] Step 5-4. Normalize the average path length of all data points and calculate the anomaly score of each data point. The calculation formula is:

[0130]

[0131] in, n Indicates the amount of data for the input sample when building the tree. c(n) is the global average path length, which is used for normalization processing. Its formula is:

[0132]

[0133] in, , is Euler's constant.

[0134] Step 6. Based on the abnormal scores of power marketing business data, a unary beta mixed model is constructed to perform adaptive abnormal group discrimination;

[0135] In order to adaptively obtain anomalies, a univariate beta mixture model is used to automatically distinguish normal values ​​and outliers in the data space. The main steps are:

[0136] Step 6-1. Use the k-means method to perform cluster analysis on the abnormal scoring results;

[0137] The anomaly score obtained in step 4 is normalized by Min-Max, so that the anomaly score is a real number between [0,1]. Then the anomaly score result is clustered using the k-means method to divide the anomaly score of the audited data into k components. For each division result, the following univariate beta modeling operation is performed.

[0138] Step 6-2. Construct a univariate beta mixture model to adaptively determine normal values ​​and outliers in the data;

[0139] In order to adaptively distinguish normal values ​​from outliers in the data, a univariate beta mixture model is constructed, and a flexible model is given to describe the outlier vector by mixing univariate beta distributions.

[0140] Univariate beta mixture distribution:

[0141]

[0142] in, It is m A univariate beta distribution with components, V i represents the anomaly score vector, and represents the parameters of the univariate beta distribution with M components; represents the mixing coefficient between the M components, and .

[0143] The probability density function of the univariate beta distribution is:

[0144]

[0145] in, represents the gamma function.

[0146] Using maximum likelihood estimation to estimate the parameters of the univariate beta mixed model, the log-likelihood function is:

[0147]

[0148]

[0149] in, represents taking the logarithm, V represents the anomaly score vector, parameter set represents the set of parameters that need to be estimated; and N Anomaly score vector V Associated, used to indicate anomaly scores V i belonging, such as Indicates i Anomaly score V i Does it belong to m components, if =1, it means it belongs to, =0 does not apply.

[0150] Use the EM algorithm to adjust the parameter set p The algorithm iterates between the expectation step and the maximization step to produce a sequence estimate , I Represents the current iteration step, and the process continues until the log-likelihood function value Stop when the threshold value stops changing.

[0151] Step 7. Identify the best univariate beta mixture model through the ICL_BIC criterion (Integrated Completed Likelihood - Bayesian Information Criterion), and use its output abnormal results as the final output abnormal results.

[0152] Through step 6, for each k For each value, a univariate beta mixed model is constructed separately. In order to identify the optimal model, the ICL-BIC criterion is used to explore the optimal model, and its calculation formula is:

[0153]

[0154] in, Represents the parameter set The estimate, Represents the belonging vector The estimate, Indicates that M The number of model parameters (i.e. parameter set length), Indicates the amount of data.

[0155] The smaller the calculated ICL-BIC value, the better the model is and the more effective it is in describing the spatial distribution of the data.

[0156] In the selected optimal model, all data are divided into M components according to the univariate beta distribution. Based on the characteristics of the abnormality score of the enhanced isolation forest algorithm, the abnormality scores of normal data points are extremely similar (or equal), so they will be divided into the same component. Because the larger the abnormality score result of the present invention, the more abnormal it is, and the smaller the score, the more normal it is, so the components with higher average abnormality scores are merged to obtain the abnormal result Outliers. The Outliers identifier takes the value of {0, 1}, where 0 represents normal and 1 represents abnormal.

[0157] Method validation

[0158] In order to verify the feasibility and effectiveness of the abnormal data active audit method proposed in this invention, an experiment based on power marketing business data was carried out, taking the most common business expansion file data in marketing audit as an example:

[0159] A sample size of 100w of power marketing business data was extracted from the database, and the specific information is shown in Table 1. Among them, there are 11 categorical variables, namely {"YHLBDM", "JLFSDM", "YDLBDM", "DJDM", "BSJFBZ", "XSJSFSDM", "XSJFBZ", "XSFTBZ", "JBDFJSFSDM", "BSFTFSDM", "LTDFJSFS"}, and 5 continuous variables, namely {"JFRL", "YDRL", "YGXSJSZ", "WGXSJSZ", "JBDFFTZ"}.

[0160] First, the classification attributes in the power business data are analyzed, and the information gain rate between the classification attributes is calculated according to step 1 and the attribute splicing is completed. By calculating the information gain rate, it is found that there is a significant correlation among the classification attributes {"YHLBDM", "JLFSDM"}, {"DJDM", "YDLBDM"}, {"BSJFBZ", "BSFTFSDM"}, among which {"DJDM", "YDLBDM"} has a significant correlation, which is consistent with the macro business law of the rule example given above that "the electricity consumption category (YDLBDM) in the customer file information is considered abnormal if it does not match the electricity price code (DJDM)". Therefore, these attributes with significant correlation are spliced ​​and encoded to form new classification attributes, and the original attributes are deleted. The present invention can not only ensure that the multi-dimensional mixed abnormal data in the data is retrieved in the subsequent steps through the splicing of related fields, but also reduce the dimension and improve the algorithm operation efficiency. The power business data set after field splicing is shown in Table 2 below.

[0161] Table 3 is the output form of the abnormal result of the audit method of the present invention. As shown in the table, the abnormal score Scores and the adaptive abnormal result Outliers are output based on the power business data. The abnormal score is the result of the abnormal coefficient calculated based on the enhanced isolation forest algorithm and normalized by Min-Max; Outliers is the identification result of the abnormality existing in the power marketing business data output based on the univariate beta mixed model (1 is abnormal, 0 is normal).

[0162] In Table 3, data anomalies 1 to 3 belong to extreme value anomalies in continuous attributes. For example, the continuous attributes of the first data "JFRL", "YDRL", and "JBDFFTZ" are all 80000, while their means are only 2.20, 40.66, and 2.69, so this data is abnormal data; and data anomalies 4 to 7 belong to multidimensional sparse anomalies in categorical attributes. For example, when "YDLBDM" is "400", "DJDM" is "4002071". After the two attributes are associated, there are only a few data in the entire data set, so it will be detected as an anomaly; the 8th data anomaly belongs to a multidimensional mixed anomaly, that is, there are anomalies in both categorical attributes and continuous attributes in the data. The above-mentioned actively identified anomalies cannot be covered by the existing audit rules, but they are indeed anomalies after being reviewed by business experts, which shows that the present invention can effectively detect multiple types of anomalies in the data.

[0163] Table 1 Power marketing business data set (partial)

[0164]

[0165] Table 2 Power marketing business data table after field splicing (part)

[0166]

[0167] Table 3 Abnormal result output of power marketing business data (partial)

[0168]

[0169] Based on this method, the present invention also provides an active audit system for abnormal business data of power marketing, which includes the following modules: a data source management module, a data reading module, a feature processing module, a model training module, an abnormal output module and an abnormal output management module.

[0170] The data source management module is used to manage the database and access information of the data required by the power marketing business data abnormality audit system, the database table where the marketing business data to be audited is located, etc.

[0171] The data reading module is used to obtain the required data from the data source management module, and perform preprocessing operations including missing value processing, deduplication, and format conversion, and save the processed data uniformly as table data required by the system.

[0172] The feature processing module is used to obtain table data from the data reading module, perform certain transformations and integration on the table data, calculate the information gain rate of the classification attribute features in the power marketing business data features, evaluate the correlation between the classification attributes and complete field splicing, and then perform feature transformation on the classification data based on the classification attribute frequency to convert it into numerical data, and generate attribute feature data that can be directly input into model training.

[0173] The model training module is used to randomly extract a certain amount or proportion of data from the data output by the feature processing module for training the enhanced isolation forest algorithm model.

[0174] The abnormal output module is used to output normal values ​​and abnormal values ​​in the power marketing business data, and the module includes:

[0175] Algorithm prediction submodule: It is used to obtain all the data output from the feature processing module based on the model trained by the model training module, input the model for prediction, and output the abnormality score of the power marketing business data;

[0176] Adaptive abnormal group discrimination submodule: Based on the abnormal score vector of power marketing business data output by the algorithm prediction submodule, for each k Value, proceed k-means Initialize and construct a singleton beta Hybrid models are used to adaptively determine normal and outliers in the data;

[0177] Abnormal result recognition submodule: It is based on multiple unary outputs of the adaptive abnormal group discrimination module beta Mixed models and unusual results, through ICL_BIC The criterion identifies the best hybrid model and uses its output anomaly results as the final output anomaly results.

[0178] The abnormal output management module is used to combine the output abnormal results with the power marketing business data output by the data source management module, and manage the final result presentation and output work.

[0179] The present invention aims at the problem that in the process of auditing power marketing business data, experts need to spend a lot of energy to formulate rules, the summarized rules have incomplete coverage, and the thresholds for abnormal assessment need to be manually defined, but abnormal data cannot be adaptively identified. A structured data unsupervised marketing data audit method that does not require empirical rule driving and supports different data types is proposed; the classification data in power marketing business data have interdependencies, and the existing unsupervised anomaly detection method only supports independent attributes, but ignores the relationship between businesses. The present invention proposes a correlation analysis and attribute processing method based on information gain rate, which can better meet the needs of power marketing audit business; the present invention aims at the problem that the classic isolation forest algorithm cannot process classified data, and proposes an improved method to transform the frequency characteristics of the classification attributes and convert the classification attributes into numerical data, so that the algorithm can be applied to mixed data. This method aims at the problem that the previous unsupervised anomaly detection method only provides anomaly degree assessment but cannot identify abnormal individuals. An abnormal result identification strategy based on k-means and univariate beta distribution is proposed, which can directly identify abnormal data.

[0180] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for actively auditing abnormal business data of power marketing, characterized in that: The following steps are involved: Step 1. For the given power marketing business data to be audited, the information gain rate is used as an evaluation index to calculate the correlation between classification attributes; Step 2. Based on the correlation analysis results between the classification attributes in the power marketing business data, the classification attributes with significant correlation are spliced ​​to generate new classification attributes, and the original classification attributes are deleted to form new mixed data; Step 3. Based on the new mixed data, perform frequency feature transformation on the categorical attributes, convert the categorical data into numerical data, and generate attribute feature data that can be directly input into model training; Step 4. Based on the attribute feature data, the enhanced isolation forest model is constructed and trained based on the enhanced isolation forest algorithm; Step 5. Based on the trained enhanced isolation forest model, calculate the anomaly score of the power marketing business data; Step 6. Based on the abnormal scores of power marketing business data, a univariate beta mixture model is constructed to perform adaptive abnormal group discrimination; Step 7. Identify the best univariate beta mixture model through the ICL_BIC criterion, and use the abnormal result output by the model as the final abnormal result; The step 1 uses the information gain rate as an evaluation index to calculate the correlation between classification attributes for the given power marketing business data to be audited, and specifically includes: Step 1-1. Calculate the information gain rate between various classification attributes in the given power marketing business data to be audited. The calculation formula of the information gain rate is: in, and They represent two different classification attributes, GainRatio(·) and Gain(·) represent the information gain rate and information gain between the classification attributes, respectively. Represents a calculated property right The information gain rate; H(·) and H(·|·) represent the calculated information entropy and conditional entropy, respectively. Indicates that the calculation is in the attribute Under the condition, the property Information entropy of v m Represents categorical attributes The mth attribute value in p(v m ) indicates that in the attribute , the calculation is equal to v m The ratio of the number of attribute values ​​to the total attribute length; Step 1-2. Convert the information gain rate between the classification attributes into the correlation between the classification attributes. The formula is as follows: in, The value range is [0,1], indicating a classification attribute and The correlation between Larger values ​​indicate greater correlation.

2. The active audit method for abnormal power marketing business data according to claim 1 is characterized in that: The step 3 performs frequency feature transformation on the categorical attributes based on the new mixed data, converts the categorical data into numerical data, and generates attribute feature data that can be directly input into model training.

3. The active audit method for abnormal business data of power marketing according to claim 2 is characterized in that: The step 4 constructs an enhanced isolation forest model based on the attribute feature data and the enhanced isolation forest algorithm. The modeling process is as follows: Step 4-1. Set the sampling number or proportion and the number of isolated trees required to build the model; Step 4-2. Sampling: Randomly extract a set number or proportion of sub-samples from the power marketing business data; for each segmentation tree, set a root node and use the root node as the current node; Step 4-3. Attribute segmentation: On the current node, randomly select several attributes as target attributes for segmentation, and construct the left subtree and right subtree of the segmentation node respectively according to the segmentation results; The segmentation strategy is as follows: in, Represents a vector consisting of several target attributes; represents the intercept vector, which is obtained by taking values ​​from a uniform distribution between the maximum and minimum values ​​of each target attribute; The normal vector, i.e. the slope of the splitting plane, is composed by randomly generating a value for each target attribute from a standard normal distribution N(0,1); Then by calculating Get the splitting hyperplane, and split the sample set of the current node based on the splitting hyperplane. If the splitting strategy of the target attribute value of the current node is less than 0, that is, it is below the splitting hyperplane, then the samples corresponding to the target attribute fall into the left subtree, and the rest fall into the right subtree; Step 4-4. Build an isolated tree; repeat step 4-3 in the child node until there is only one data in the child node or it is impossible to continue cutting or the child node has reached the maximum depth of the set tree, then stop the step and complete the construction of the isolated tree; Step 4-5. Construct an enhanced isolation forest; according to the sampling number or ratio set in step 4-1 and the number of trees to be constructed, loop steps 4-2, 4-3, and 4-4 to complete the construction of all isolated trees to form an enhanced isolation forest.

4. The active audit method for abnormal power marketing business data according to claim 3 is characterized in that: In step 5, the abnormality score of the power marketing business data is calculated based on the trained enhanced isolation forest model, which specifically includes the following sub-steps: Step 5-1. Input all the data to be audited into the enhanced isolation forest model. For each data point, after passing through the enhanced isolation forest model composed of T trees, T segmentation results will be obtained; Step 5-2. For each data point, calculate the path length h(x) of the data point in each isolated tree, that is, the number of edges that the data point passes through from the root node of the tree to the leaf node; Step 5-3. Calculate the average path length of each data point in the enhanced isolation forest, that is, in all trees. The calculation formula is: Where E(h(x)) represents the average path length, T represents the number of trees in the enhanced isolation forest, and h t (x) represents the path length of data point x in the tth tree; Step 5-4. Normalize the average path length of all data points and calculate the anomaly score of each data point. The calculation formula is: Among them, n represents the amount of data of the input sample when building the tree, and c(n) is the global average path length, which is used for normalization processing. The formula is: c(n)=2H(n-1)-2(n-1) / n Among them, H(k)=ln(k)+ε, ε=0.5772156649, ε is the Euler constant.

5. The active audit method for abnormal power marketing business data according to claim 4 is characterized in that: In step 6, based on the abnormal scores of the power marketing business data, a univariate beta mixture model is constructed to perform adaptive abnormal group discrimination, specifically including: Step 6-1. Use the k-means method to perform cluster analysis on the abnormal score results, and divide the abnormal scores of the power marketing business data to be audited into k components; Step 6-2. Construct a univariate beta mixture model and output the univariate beta mixture model and abnormal results.

6. The method for actively auditing abnormal business data of power marketing according to claim 5, characterized in that: The univariate beta mixed model includes: Univariate beta mixture distribution: Among them, B m (V i |α, β) is the univariate beta distribution of the mth component, V i represents the anomaly score vector, α={α1,…,α M } and β = {β1, ... β M } represents the parameters of the univariate beta distribution of M components; λ = {λ1, …, λ M } represents the mixing coefficient between M components, and The probability density function of the univariate beta distribution is: Where Γ(·) represents the gamma function; Using maximum likelihood estimation to estimate the parameters of the univariate beta mixed model, the log-likelihood function is: Where log(·) represents the logarithm, V represents the anomaly score vector, and the parameter set p = {λ1,…,λ m ,α1,…,α m , β1,…,β m } represents the set of parameters to be estimated; η={η1,…,η N } is associated with N anomaly score vectors V, indicating the anomaly score V i The belonging of im represents the i-th anomaly score V i Whether it belongs to the mth component, if η im =1, it means that it belongs to, η im =0, then it does not belong.

7. The active audit method for abnormal power marketing business data according to claim 6 is characterized in that: In step 7, the ICL-BIC criterion is used to find the optimal model, and the calculation formula is: in, represents the estimate of the parameter set P, represents the estimation of the belonging vector η, Q M represents the number of parameters of a model with M components, and N represents the amount of data.

8. An active auditing system for abnormal power marketing business data applied to an active auditing method for abnormal power marketing business data according to any one of claims 1 to 7, characterized in that: include: Data source management module, data reading module, feature processing module, model training module, abnormal output module and abnormal output management module; The data source management module is used to manage the database and access information of the data required by the power marketing business data abnormality audit system, and the database table where the marketing business data to be audited is located; The data reading module is used to obtain the required data from the data source management module, and perform preprocessing operations including missing value processing, deduplication, and format conversion, and save the processed data uniformly as table data required by the system; The feature processing module is used to obtain table data from the data reading module, transform and integrate the table data, calculate the information gain rate of the classification attribute features in the power marketing business data features, evaluate the correlation between the classification attributes and complete the field splicing, and then transform the classification data into numerical data based on the classification attribute frequency to generate attribute feature data that can be directly input into model training; The model training module is used to randomly extract a preset number or proportion of data from the data output by the feature processing module for training the enhanced isolation forest algorithm model; The abnormal output module is used to output normal values ​​and abnormal values ​​in the power marketing business data; The abnormal output management module is used to combine the output abnormal results with the power marketing business data output by the data source management module, and manage the final result presentation and output work.

9. The active audit system for abnormal business data of power marketing according to claim 8, characterized in that: The abnormal output module comprises: Algorithm prediction submodule: It is used to obtain all the data output from the feature processing module based on the model trained by the model training module, input the model for prediction, and output the abnormality score of the power marketing business data; Adaptive abnormal group discrimination submodule: Based on the vector of abnormal scores of power marketing business data, for each k value within the specified range, k-means is initialized and a univariate beta mixture model is constructed to adaptively determine normal values ​​and abnormal values ​​in the data; The abnormal result identification submodule is used to identify the best mixed model based on the multiple univariate beta mixed models and abnormal results output by the adaptive abnormal group discrimination module through the ICL_BIC criterion and use the abnormal result output by the mixed model as the final abnormal result.

Images