Reasonability evaluation analysis and optimization method suitable for business data strategy threshold

Abstract

The invention discloses a method suitable for evaluating, analyzing and optimizing the reasonability of a business data strategy threshold. The method comprises two parts of data feature processing and algorithm operation. The data feature processing part cleans and converts the extracted sample data, and then performs feature selection, binary classification value replacement, feature association, feature grouping and source data grouping identification to form algorithm operation source data. The accuracy rate and the recall rate of the model are comprehensively considered in algorithm operation, so that the reasonability of a strategy threshold value is evaluated, and an optimal suggested configuration threshold value closest to a balance point is calculated. According to the invention, intelligent analysis and evaluation can be carried out on the operation data and optimization suggestions can be given.

Description

Technical Field

[0001] The present invention relates to the field of enterprise informatization technology, and in particular to a method for evaluating, analyzing and optimizing the rationality of business data strategy thresholds. Background Art

[0002] Business rules in enterprise information-based business support systems are often expressed using various strategies. For example, business data preprocessing strategies are used to remove duplicate business data, handle missing values, and perform numerical conversions. Data risk identification strategies are used to identify and intercept risks in data content. These strategy rules require multiple thresholds to calculate binary classification results for the data, such as whether the data contains risks (yes / no).

[0003] During system construction and operation, as the volume of enterprise business data grows, the scope of business expands, and the complexity of business increases, these manually summarized, refined, and configured policy rules require periodic rule execution effectiveness evaluation and policy optimization to prevent the blunting of policy rule execution effectiveness. However, currently, the analysis and evaluation of policy rule execution effectiveness and the adjustment of policy configuration thresholds that require improvement generally involve system operators regularly writing data statistics scripts based on operational data in the database. The statistical results are then manually compared and analyzed to derive policy evaluation results, and more reasonable policy thresholds are then estimated based on experience. This approach suffers from two significant and common problems: 1. Low data analysis efficiency and non-objective evaluation results; 2. The accuracy of the recommended policy thresholds derived from experience is low, making it difficult to adjust business rules in a timely and accurate manner. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for evaluating, analyzing and optimizing the rationality of business data policy thresholds.

[0005] The technical solution adopted in the present invention is:

[0006] A method for evaluating and optimizing the rationality of business data policy thresholds, including the following steps:

[0007] Step 1: Clean the extracted business data sample data and extract features;

[0008] Step 2: Select detailed features of policy rules, which include primary variables and secondary variables, and identify the main factors of the business data interception condition calculation process data according to the configuration;

[0009] Specifically, taking call bill data as an example, the service data interception condition details have a "Is it a primary cause" field (configured as 1 or 0). The calculation process data refers to the service data interception condition calculation log, which records the left-hand side calculation results, right-hand side calculation results, operators, and other data of the condition details during the interception process. The meaning of the primary cause is, for example, the condition in the rule "Fuzhou voice call bill charges greater than 100 yuan will not be combined" will be specifically configured as three condition details: (1) Call bill local network = Fuzhou; (2) Call bill type = voice; (3) Call bill charge > 100 yuan. Among them, the condition details related to the threshold are the primary cause conditions, that is, condition detail (3).

[0010] Step 3: Convert the audit results of the audit data and the confirmation results of the alarm data in the intercepted business data into binary classification result data respectively;

[0011] Specifically, the audit results of intercepted business data and the confirmation results of alarm data are converted and marked, that is, the results are converted into two categories (0 for releasing the behavior and 1 for confirming the interception; 0 for ignoring the alarm and 1 for confirming the alarm);

[0012] Step 4: Use the business data unique identifier to associate each piece of business data and business data interception condition calculation process data with the corresponding intercepted business data review data and alarm data;

[0013] Step 5: Classify the selected policy rule details and group the associated data by sub-variables to generate grouped data;

[0014] Step 6: assign corresponding group labels to the associated business data, business data interception calculation process data, and their corresponding intercepted business data review data according to the secondary variable grouping result;

[0015] Step 7: Generate basic data table records including binary classification operation source data table and grouping messages;

[0016] The algorithm operation part includes two parts: "Evaluating the rationality of the current threshold" and "Calculating the recommended configuration threshold". It specifically includes the following steps:

[0017] Step 8: Evaluate the rationality of the current threshold: Using the current policy rule threshold configuration as the prediction threshold, calculate the precision and recall of the data samples in the binary classification operation source data table. Then, calculate the absolute difference value (ADV) based on the precision and recall, and assign a score based on the configuration evaluation (scoring formula: 100 – ADV * 100).

[0018] Step 9, calculate the recommended configuration threshold: sort the main variable values of all binary operation source data tables for feature extraction, and calculate the precision (Precision), recall (Recall) and absolute difference (ADV) again as the prediction threshold. Sort the ADV values of all prediction thresholds from small to large to obtain the optimal prediction threshold closest to the balance point (BEP, Break_eventPoint, at this time ADV = 0).

[0019] Specifically, in the example above, for the primary variable value, such as "Fuzhou voice call bill charges are greater than 100 yuan," the threshold involved is the call bill charge. This means the charges of all intercepted call bills are used as the prediction threshold.

[0020] Furthermore, the cleaning transformation in step 1 includes filling in missing values, removing abnormal data and duplicate data, and normalizing data.

[0021] Furthermore, in step 2, the primary variable is the business data amount, and the secondary variable is the business data type;

[0022] Furthermore, before calculating the precision (Precision) and recall (Recall), the data samples in the binary classification operation source data table are divided into four categories: true positive examples TP, false positive examples FP, positive negative examples TN and false negative examples FN.

[0023] Furthermore, precision refers to the proportion of samples predicted to be positive that are actually positive. The specific calculation formula is expressed as:

[0024]

[0025] Specifically, TP stands for True Positive (should be intercepted and correctly intercepted, and should be intercepted means that the interception is confirmed after manual review in the sample); FP stands for False Positive (should be released but was incorrectly intercepted, and should be released means that the sample needs to be released after manual review). Simply pursuing accuracy will result in a strategy with few interceptions (in this case, FP is low, that is, the strategy execution goal is to minimize interceptions).

[0026] Furthermore, recall, or hit rate, refers to the ratio of the number of samples correctly predicted as positive examples to all true positive examples. The specific calculation formula is:

[0027]

[0028] Specifically, FN represents a false negative (a violation that should not be released but is mistakenly released); simply pursuing recall will result in excessive interception by the strategy (in this case, FN is low, that is, the strategy execution goal is to intercept as much as possible).

[0029] It should be noted that precision and recall are a set of contradictory metrics. When precision is high, the FP value is small, but the Recall value is often low, that is, the FN value is large; when the Recall value is large (that is, the FN value is small), the precision is often low, that is, the FP value is large.

[0030] Furthermore, the Absolute Difference Value (ADV) is used to express the degree of deviation between precision and recall. The specific calculation formula is:

[0031] ADV=|Precision-Recall|.

[0032] The present invention adopts the above technical solution, which can accurately analyze and evaluate the policy rule thresholds and provide optimization suggestions; it can reduce the difficulty of manual operation and maintenance of the information system, reduce workload, and improve operational efficiency and quality; the present invention can solve the problems of "low data analysis efficiency and non-objective evaluation results", "low accuracy of recommended policy thresholds based on experience, and business rules cannot be adjusted in a timely and accurate manner", etc., and can perform intelligent analysis and evaluation of operational data and provide optimization suggestions. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0034] Figure 1 Schematic diagram of the process of data feature processing of the present invention;

[0035] Figure 2 Schematic diagram of the relationship curve between the precision rate, recall rate and absolute difference index and the prediction threshold of the present invention;

[0036] Figure 3 Schematic diagram of the flow of the algorithm operation of the present invention;

[0037] Figure 4 Schematic diagram of the specific process for calculating the precision, accuracy and ADV value of the prediction threshold. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0039] like Figures 1 to 4As shown in Figure 1, the present invention discloses a method for evaluating and analyzing the rationality of business data policy thresholds and optimizing the method, which can improve the efficiency of data analysis of policy rule execution results, improve the accuracy of policy rule threshold rationality evaluation, the timeliness of optimization, and the accuracy of recommended correction configuration thresholds. The information processing method mainly includes two steps: data feature processing and algorithm operation. In the algorithm operation step, the precision (Precision) and recall (Recall) of the model are comprehensively considered to evaluate the rationality of the policy threshold and calculate the optimal recommended configuration threshold closest to the equilibrium point (BEP, Break_eventPoint).

[0040] Specifically, the data feature processing part: after the extracted sample data is cleaned and converted (missing value filling, data normalization processing), feature selection, binary value replacement, feature association, feature grouping and source data group identification are performed to form the algorithm operation source data. Each group is a target unit for analysis, evaluation and optimization, such as the following call bill consolidation policy rule "When the user's voice call bill amount is greater than 100 yuan and the Internet call bill amount is greater than 80 yuan, the account consolidation action is intercepted and the account can only be consolidated after manual review." Contains two analysis units: "Voice call bill amount is greater than 100 yuan" and "Internet call bill amount is greater than 80 yuan". This policy rule needs to extract configuration data (call bill consolidation policy rule details) and the following operational data for feature processing: 1. Call bill data; 2. Call bill interception condition calculation process data; 3. Alarm data; 4. Intercepted call bill review data. The specific steps of the present invention are as follows:

[0041] Step 1: Clean the extracted business data sample data and extract features;

[0042] Furthermore, the cleaning transformation includes filling in missing values, removing abnormal data and duplicate data, and normalizing data.

[0043] Step 2: Select the detailed features of the policy rules. The detailed features of the policy rules include primary variables and secondary variables. According to the configuration, the main cause of the business data interception condition calculation process data is identified to clarify whether the corresponding business data interception condition details are the main cause;

[0044] Specifically, the service data interception condition details have a field for "Is it the main cause" (configured as 1 or 0); the meaning of the main cause, such as the description of "Fuzhou voice call bill fee is greater than 100 yuan", will be configured as three condition details: (1) Call bill local network = Fuzhou; (2) Call bill type = voice; (3) Call bill fee > 100 yuan; among them, the condition details related to the threshold are the main cause conditions, that is, condition detail (3).

[0045] Calculation process data refers to the business data interception condition calculation log, which records the left value calculation result, right value calculation result, operator and other data of the condition details during the interception process.

[0046] Furthermore, in step 2, the primary variable is the business data amount, and the secondary variable is the business data type;

[0047] Step 3: Convert the audit results of the audit data and the confirmation results of the alarm data in the intercepted business data into binary classification result data respectively;

[0048] Specifically, the audit results of intercepted business data and the confirmation results of alarm data are converted and marked, that is, the results are converted into two categories (0 for releasing the behavior and 1 for confirming the interception; 0 for ignoring the alarm and 1 for confirming the alarm);

[0049] Step 4: Use the business data unique identifier to associate each piece of business data and business data interception condition calculation process data with the corresponding intercepted business data review data and alarm data;

[0050] Step 5: Classify the selected policy rule details and group the associated data by sub-variables to generate grouped data;

[0051] Step 6: assign corresponding group labels to the associated business data, business data interception calculation process data, and their corresponding intercepted business data review data according to the secondary variable grouping result;

[0052] Step 7: Generate basic data table records including binary classification operation source data table and grouping messages;

[0053] The algorithm operation part includes two parts: "Evaluating the rationality of the current threshold" and "Calculating the recommended configuration threshold". It specifically includes the following steps:

[0054] Step 8: Evaluate the rationality of the current threshold: Using the current policy rule threshold configuration as the prediction threshold, calculate the precision and recall of the data samples in the binary classification operation source data table. Then, calculate the absolute difference value (ADV) based on the precision and recall, and assign a score based on the configuration evaluation (scoring formula: 100 – ADV * 100).

[0055] Step 9, calculate the recommended configuration threshold: sort the main variable values of all binary operation source data tables for feature extraction, and calculate the precision (Precision), recall (Recall) and absolute difference (ADV) again as the prediction threshold. Sort the ADV values of all prediction thresholds from small to large to obtain the optimal prediction threshold closest to the balance point (BEP, Break_eventPoint, at this time ADV = 0).

[0056] Specifically, in the example above, for the primary variable value, such as "Fuzhou voice call bill charges are greater than 100 yuan," the threshold involved is the call bill charge. This means the charges of all intercepted call bills are used as the prediction threshold.

[0057] Furthermore, before calculating the precision and recall, the binary classification operation source data table data samples are divided into four categories: true positive examples TP, false positive examples FP, positive negative examples TN, and false negative examples FN:

[0058] 1) TP, representing true positives (True Positive, which should be intercepted and correctly intercepted. The interception should be confirmed after manual review in the sample);

[0059] 2) FP, stands for False Positive (False Positive, which should be released but is mistakenly intercepted. The sample that should be released needs to be released after manual review);

[0060] 3) TN, indicating True Negative (should be released and correctly released);

[0061] 4) FN, stands for False Negative (false negative, which should not be released but is mistakenly released);

[0062] Furthermore, precision refers to the proportion of samples predicted to be positive that are actually positive, which can be expressed by the formula:

[0063]

[0064] Specifically, TP stands for True Positive (should be intercepted and correctly intercepted, and should be intercepted means that the interception is confirmed after manual review in the sample); FP stands for False Positive (should be released but was incorrectly intercepted, and should be released means that the sample needs to be released after manual review). Simply pursuing accuracy will result in a strategy with few interceptions (in this case, FP is low, that is, the strategy execution goal is to minimize interceptions).

[0065] Furthermore, recall, or hit rate, refers to the ratio of the number of samples correctly predicted as positive to all true positive examples, which can be expressed by the formula:

[0066]

[0067] Specifically, FN represents a false negative (a violation that should not be released but is mistakenly released); simply pursuing recall will result in excessive interception by the strategy (in this case, FN is low, that is, the strategy execution goal is to intercept as much as possible).

[0068] It should be noted that precision and recall are contradictory metrics. When precision is high, the FP value is small, but the recall value is often low, that is, the FN value is large. When the recall value is large (that is, the FN value is small), the precision is often low, that is, the FP value is large. This paper uses the absolute difference value (ADV) to express the degree of deviation between precision and recall:

[0069] ADV=|Precision-Recall|

[0070] For example, let's configure a new call bill consolidation policy rule in the billing system: "When a user's single voice call bill exceeds 100 yuan, the consolidation action is blocked and can only be performed after manual review." The sample data exported from the actual production system is sorted from smallest to largest by the primary variable value (call bill amount, unit: cents). The precision, recall, and ADV of all corresponding instances are calculated as prediction thresholds. The relationship between the calculation results and the prediction threshold is as follows: Figure 2 As shown in the curve, we can see that as the threshold increases, the precision gradually increases, that is, the FP value gradually decreases (the number of instances that should be released but are incorrectly intercepted gradually decreases); the recall gradually decreases, that is, the FN value gradually increases (the number of instances that should not be released but are incorrectly released gradually increases).

[0071] There is an intersection point between the precision and recall curves. Precision calls this point the break-event point (BEP). The goal of the algorithm is to find the prediction threshold that is closest to the break-event point.

[0072] The present invention can improve the efficiency of data analysis of policy rule execution results, improve the accuracy of assessment of the rationality of policy rule thresholds, the timeliness of optimization, and the accuracy of recommended corrections to configuration thresholds. The information processing method mainly includes two steps: "data feature processing" and "algorithm operation". In the algorithm operation step, the precision (Precision) and recall (Recall) of the model are comprehensively considered to evaluate the rationality of the policy threshold and calculate the optimal recommended configuration threshold closest to the equilibrium point (BEP, Break_event Point). The present invention can solve the problems of "low data analysis efficiency and non-objective evaluation results", "low accuracy of recommended policy thresholds derived from experience, and inability to adjust business rules in a timely and accurate manner", and can perform intelligent analysis and evaluation of operational data and provide optimization suggestions.

[0073] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

Claims

1. A method for evaluating, analyzing, and optimizing the rationality of business data policy thresholds, characterized by: The specific steps are as follows: Step 1: Clean the extracted business data sample data and extract features; Step 2: Select the detailed features of the policy rules. The detailed features of the policy rules include primary variables and secondary variables. According to the configuration, the main cause of the business data interception condition calculation process data is identified to clarify whether the corresponding business data interception condition details are the main cause; Step 3: Convert the audit results of the audit data and the confirmation results of the alarm data in the intercepted business data into binary classification result data respectively; Step 4: Use the business data unique identifier to associate each piece of business data and business data interception condition calculation process data with the corresponding intercepted business data review data and alarm data; Step 5: Classify the selected policy rule details and group the associated data by sub-variables to generate grouped data; Step 6: assign corresponding group labels to the associated business data, business data interception calculation process data, and their corresponding intercepted business data review data according to the secondary variable grouping result; Step 7: Generate basic data table records including binary classification operation source data table and grouping messages; Step 8: Evaluate the rationality of the current threshold: Use the current policy rule threshold configuration as the prediction threshold, calculate the precision and recall of the data samples in the binary classification operation source data table, and then calculate the absolute difference ADV based on the precision and recall to obtain the evaluation score of the current policy rule threshold configuration; Step 9: Calculate the recommended configuration threshold: Sort the main variable values of all binary classification operation source data tables for feature extraction, and recalculate the precision, recall, and absolute difference ADV as the prediction threshold. Sort the ADV values of all prediction thresholds from small to large to obtain the optimal prediction threshold closest to the balance point.

2. A method for evaluating, analyzing, and optimizing the rationality of business data policy thresholds according to claim 1, characterized in that: The cleaning transformation in step 1 includes filling in missing values, removing abnormal data and duplicate data, and normalizing data.

3. The method for evaluating, analyzing, and optimizing the rationality of business data policy thresholds according to claim 1, characterized in that: In step 2, the primary variable is the business data amount, and the secondary variable is the business data type.

4. The method for evaluating, analyzing, and optimizing the rationality of business data policy thresholds according to claim 1, wherein: Before calculating the precision and recall, the data samples in the binary classification operation source data table are divided into four categories: true positive examples TP, false positive examples FP, positive and negative examples TN, and false negative examples FN. Among them, true positive examples TP indicate that they should be intercepted and are correctly intercepted. Interception means that the sample is confirmed to be intercepted after manual review. False positive examples FP indicate that they should be released but are mistakenly intercepted. Release means that the sample needs to be released after manual review. Positive and negative examples TN indicate that they should be released and are correctly released. False negative examples FN indicate that they should not be released but are mistakenly released.

5. The method for evaluating, analyzing, and optimizing the rationality of business data policy thresholds according to claim 4, characterized in that: The specific calculation formula of precision is expressed as: Among them, TP represents true positive examples and FP represents false positive examples.

6. The method for evaluating, analyzing, and optimizing the rationality of business data policy thresholds according to claim 4, characterized in that: The specific calculation formula of recall rate Recall is expressed as: Among them, TP represents true positive examples and FN represents false negative examples.

7. The method for evaluating, analyzing, and optimizing the rationality of business data policy thresholds according to claim 1, characterized in that: The specific calculation formula of the absolute difference ADV is expressed as: ADV=|Precision-Recall| Among them, Precision represents precision and Recall represents recall.

Images