Policy evaluation device and policy evaluation method

Abstract

To provide a measure evaluation apparatus and a measure evaluation method capable of making a preliminary evaluation of how effective each accumulated know-how is in the environment to which the know-how is applied, and capable of contributing from an economic standpoint, for example.SOLUTION: A measure evaluation apparatus is configured to: extract suggestions for business improvement from business performance data in a first environment; generate measures to improve business indexes based on the extracted suggestions; evaluate the effectiveness of measures in the first environment; accumulate the business improvement effects of the measures in the first environment and data including evaluation conditions when the business improvement effects are evaluated as a rule of experience for the first environment, regarding the measures for which evaluation is effective; extract data that satisfies the evaluation conditions from the business performance data in a second environment; calculate the predicted value of the business improvement effect in the second environment and calculate the reliability by evaluating the consistency of the evaluation conditions of the business performance data in the first environment and the second environment, by using the extracted data; and display the predicted value and reliability of the business improvement effects.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a policy evaluation device and a policy evaluation method.

Background Art

[0002] In various fields, a method of accumulating know-how for improving the performance of a target business by analyzing business performance data accumulated in a business system has been attracting attention.

[0003] In Patent Document 1, not only focusing on factors strongly related to indicators (objective variables) related to performance, but also on explanatory variables with high correlation with the objective variables, and based on the noticed feature quantities obtained by analyzing data, generating measures from a new perspective, and evaluating the effectiveness using the data after the measure is implemented in the business, a technique for storing new know-how in a knowledge base is described.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technique of Patent Document 1, not only know-how obtained by analyzing business performance data, but also know-how from a new perspective whose effectiveness has been confirmed by implementing and evaluating measures generated from a new perspective can be accumulated. On the other hand, the know-how thus obtained can be expected to be effective even when implementing subsequent measures in the environment where the measure was implemented and its effect was confirmed, but its effectiveness is unclear in different similar environments (for example, the business is the same but the place of residence of the measure implementation target is different, etc.).

[0006] Therefore, the present invention aims to provide a policy evaluation device and policy evaluation method that can pre-evaluate how effective each accumulated piece of know-how is in an environment different from the one in which it was accumulated, and that can contribute, for example, from an economic standpoint, when it is assumed that know-how will be applied to an environment different from the one in which it was accumulated. [Means for solving the problem]

[0007] To solve the above problems, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above objectives, but as one example, a policy evaluation device in the following embodiment is provided. This policy evaluation device comprises a suggestion extraction unit, a policy evaluation unit, an empirical rule storage unit, an empirical measurement pre-evaluation unit, and an information output unit. The suggestion extraction unit extracts suggestions regarding business improvement from business performance data that stores the results of business in a first environment. The policy generation unit generates policies to improve business indicators based on the extracted suggestions. The policy evaluation unit evaluates the effectiveness of the policies in the first environment. The empirical rule storage unit stores the policies that were effective in the policy evaluation unit as empirical rules. The empirical measurement pre-evaluation unit pre-evaluates the effectiveness of the empirical rules in a second environment different from the first environment, based on business performance data in the second environment. The information output unit is used for outputting information. The empirical rule storage unit stores data including the business improvement effect of the policies in the first environment and the evaluation conditions when evaluating the business improvement effect as empirical rules for the first environment. The empirical rule pre-evaluation unit extracts data that satisfies the evaluation conditions from the business performance data in the second environment when pre-evaluating the effectiveness of the empirical rule in the first environment. Using the extracted data, the empirical rule pre-evaluation unit calculates a predicted value of the business improvement effect in the second environment and calculates the reliability, which is the pre-evaluation value of the empirical rule, by evaluating the consistency of the evaluation conditions of the business performance data in the first and second environments. The information output unit displays the predicted value of the business improvement effect and the reliability.

[0008] Furthermore, a policy evaluation method in the following manner is provided. This policy evaluation method uses a computing device, a storage unit, and a display unit. The computing device extracts suggestions for business improvement from business performance data stored in the storage unit, which contains the actual business performance in a first environment. Based on the extracted suggestions, it generates policies to improve business indicators. It evaluates the effectiveness of the policies in the first environment. For policies that were evaluated as effective, it stores data in the storage unit as an empirical rule for the first environment, including the business improvement effect of the policies in the first environment and the evaluation conditions used when evaluating the business improvement effect. It extracts data that satisfies the evaluation conditions from the business performance data in the second environment stored in the storage unit. Using the extracted data, it calculates a predicted value of the business improvement effect in the second environment. It also evaluates the consistency of the evaluation conditions of the business performance data in the first and second environments to calculate reliability, which is a pre-evaluation value of the empirical rule, and displays the predicted value of the business improvement effect and reliability on the display unit. [Effects of the Invention]

[0009] According to the present invention, when considering the application of accumulated know-how to different environments, it becomes possible to evaluate its effectiveness in advance, thereby efficiently supporting decisions regarding the implementation of measures in the target environment, and contributing, for example, from an economic standpoint. Further issues, configurations, and effects not mentioned above will be clarified by the following description of embodiments for carrying out the invention. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the hardware and software configuration of a policy evaluation device. [Figure 2] This diagram shows a basic flow chart of an example of a policy evaluation process. [Figure 3] This figure shows an example of business performance data. [Figure 4] This figure shows an example of an empirical rule table. [Figure 5] This figure shows an example of the processing flow for pre-evaluation. [Figure 6]This figure shows an example of a table summarizing the results of the preliminary evaluation. [Figure 7] This diagram illustrates an example of a process for performing a preliminary evaluation. [Figure 8] This figure shows an example of a business model that undergoes state transitions. [Figure 9] This figure shows an example of the results obtained by verifying the effect of a specific empirical rule in the case of Figure 8. [Figure 10] This figure illustrates an example of a pre-evaluation in a different environment than that shown in Figure 9. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described in detail below with reference to the drawings. The embodiments are illustrative examples for illustrating the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. Examples of various types of information may be described using expressions such as "table," but these types of information may also be represented by other data structures. For example, various types of information such as "XX table" may also be referred to as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable. In this embodiment, we describe a policy evaluation device that, when considering the application of know-how accumulated in one environment (referred to as Environment 1) to another different environment (referred to as Environment 2), pre-evaluates the effectiveness of that know-how in Environment 2 using business performance data from Environment 2. Note that in each figure, similar components may be denoted by the same reference numerals.

[0012] Figure 1 shows an example of the hardware and software configuration of the policy evaluation device 101. The policy evaluation device 101 comprises a main memory 102, a secondary memory 104, an arithmetic unit 105, a network interface 106, and an input / output interface 107 as its hardware configuration. Each hardware component is connected to the others via an internal bus. The storage unit can, for example, include a main memory 102 and a secondary memory 104.

[0013] The main memory 102 is a type of memory such as DRAM (Dynamic Random Access Memory) and stores the program executed by the arithmetic unit 105 and the information used by the program. The main memory 102 may be composed of volatile or non-volatile memory elements. The program and information stored in the main memory 102 will be described in detail later.

[0014] The secondary storage device 104 is an HDD (Hard Disk Drive) or SSD (Solid State Drive), etc., and permanently stores data. Programs and information stored in the main memory 102 may also be stored in the secondary storage device 104. In this case, the arithmetic unit 105 reads the programs and information from the secondary storage device 104, loads them into the main memory 102, and executes the loaded programs. The arithmetic unit 105 is a processor, GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), etc., and executes programs stored in the main memory 102. The network interface 106 (NW / IF in Figure 1) communicates with external devices via a network. The input / output interface 107 (Input / Output / IF in Figure 1) is a device for inputting data such as a keyboard, mouse, and touch panel, and a device for outputting or displaying data such as a display, and displays various analysis and evaluation results, and accepts various input operations from the user. The input / output interface 107 constitutes an operation unit used by the user for operation and a display unit used for displaying information.

[0015] The policy evaluation device 101 includes, as software modules, a control unit 103, a suggestion extraction unit 108, a policy generation unit 109, a policy evaluation unit 110, a rule of thumb accumulation unit 111, a rule of thumb pre-evaluation unit 112, an information input unit (not shown), and an information output unit (not shown). These program entities are stored in the main memory device 102 or the auxiliary memory device 104, loaded when the device is started, and realize functions as data processing execution modules.

[0016] The control unit 103 controls the operations of various software modules of the policy evaluation device 101. In the following description of the processing, unless otherwise specified, the control unit 103 controls the processing. The suggestion extraction unit 108 performs data analysis processing on the input data and extracts data trends (referred to as suggestions here) that lead to performance improvement of the target business. The policy generation unit 109 performs processing to generate policies to be implemented for the target business. As an example of policy generation, the policy generation unit 109 automatically generates policies that satisfy the suggestions based on the suggestions obtained from the suggestion extraction unit 108. The policy evaluation unit 110 performs processing to quantitatively evaluate the effect of the policies implemented in each environment on the business performance from the business performance data after the implementation of the policies. The rule of thumb accumulation unit 111 performs processing to accumulate, as rules of thumb, the policies determined as the know-how to be accumulated for each of the policies whose effects have been quantitatively evaluated by the policy evaluation unit 110. The rule of thumb pre-evaluation unit 112 performs processing to pre-evaluate the effectiveness of the accumulated rules of thumb using the business performance data in the environment to which the rules of thumb are applicable. The information input unit is used for inputting information from the user. The information output unit outputs information such as the result of the pre-evaluation (for example, display output) and is used to present information to the user. Details of the operations of these software modules will be described later.

[0017] As data managed by the policy evaluation device 101, there are business performance data (environment 1) 1021, rule-of-thumb table (environment 1) 1022, business performance data (environment 2) 1023, and rule-of-thumb table (environment 2) 1024. Note that these data may be saved in the secondary storage device 104 after being calculated on the main storage device 102. The business performance data (environment 1) 1021 stores past performance data of the operations carried out in environment 1. The rule-of-thumb table (environment 1) 1022 stores data that has been actually implemented as policies in environment 1 and accumulated as rules of thumb. The business performance data (environment 2) 1023 stores past performance data of the operations carried out in environment 2. The rule-of-thumb table (environment 2) 1024 stores data that has been actually implemented as policies in environment 2 and accumulated as rules of thumb. Details of these data will be described later.

[0018] Next, using the processing flow in FIG. 2, the specific processing of the present invention will be described in detail, which accumulates the rules of thumb in environment 1 from the business performance data of environment 1 input to this device and performs a pre-evaluation of effectiveness when applying the accumulated rules of thumb to environment 2. FIG. 2 shows the basic flow of an example of the policy evaluation process. Here, as an example of the operation used in the explanation, the operation is to make a telephone solicitation for a fiber-optic Internet contract for residential use to customer users who have already contracted a mobile phone line. The business indicator KPI (Key Performance Indicator) in this business is assumed to be the conversion rate (the ratio of the number of contracts made to the number of solicitations) based on the conversion result of whether a fiber-optic contract has been reached by the telephone solicitation.

[0019] First, the process of accumulating the rules of thumb in environment 1 will be described. That is, a series of processes (S201 to S207) from data input in environment 1 to updating the rule-of-thumb table will be described.

[0020] Processing in this device is initiated, and the latest business performance data (Environment 1) 1021 at that time is input and saved to each storage device (S201). Figure 3 shows an example of business performance data. This data stores the ID of the operator who conducted the solicitation, the ID of the customer who was solicited, the content of the solicitation, and whether or not the solicitation resulted in a sale for each solicitation operation performed for each timestamp. Attribute information linked to the operator ID is stored as shown in the operator table 302. Similarly, attribute information linked to the customer ID is stored as shown in the customer table 303. Analysis processing is performed on the input business data to extract insights that suggest a high correlation with the KPI (S202). Various known techniques can be applied in this analysis processing, but one example is to find explanatory variables that have a high correlation with the dependent variable when the KPI is the dependent variable.

[0021] Next, measures to be taken for operations in this environment are generated (S203). Various known techniques can be applied to the method of generating the measures to be taken, but one example is to generate measures that change the explanatory variables in a way that improves the KPI, based on the data trends obtained from the aforementioned suggestions (explanatory variables that have a high correlation with the dependent variable). Alternatively, the user of this device (the person who decides whether to implement the measures) may create the measures to be taken by referring to the aforementioned suggestions.

[0022] From the multiple proposed measures generated as described above, the user decides which measures to actually implement in the target business and then implements them (S204). This device manages and stores information about the measures that were actually implemented, the ID of the customer user to whom the measures were implemented, and various conditions at the time of implementation (such as the date and time and the operator's ID). After the results of each measure implemented in Environment 1 are saved as new business data, the business data (Environment 1) is updated to the latest data (S205).

[0023] Using updated business data, the impact of the implemented measures on KPIs will be evaluated by comparing the KPI values ​​of the target group that implemented the measures with the KPI values ​​of the control group (S206). Specifically, the evaluation method will involve designating the target group that implemented the measures as Group A (a certain number of targets, 50 in this case) and the target group that has not yet implemented the measures as Group B (also 50), and conducting an A / B test. To evaluate the impact of the implemented measures on KPIs, a statistical test will be performed on the KPI index values ​​between each group, and if a significant difference is found, it will be concluded that the measures did indeed have an impact on the KPIs.

[0024] For measures that have been evaluated and whose impact on KPIs has been confirmed, these measures can be considered highly reliable know-how in this environment, and their information is stored in the rules of thumb table (Environment 1) 1022 (S207). The criteria for selecting measures to be stored in the rules of thumb table from among the implemented measures may be automatically determined based on the aforementioned statistical significance and the magnitude of the impact on KPIs, or it may be configured so that the user can select them.

[0025] Figure 4 shows an example of an accumulated rule of thumb table. Rule of thumb table 401 is an example where, for effective measures, the evaluation conditions of the A / B test used to evaluate the effectiveness of the measures are accumulated in addition to the details of the measures themselves. In this table, for each rule of thumb ID, the conditions of the customer users who are the target of the measures (measure target conditions), the measures action to be taken, the KPI effect amount as the impact of this measure on the KPI indicator, and the likelihood of this rule of thumb being true are recorded.

[0026] The KPI effect size is the difference between the conversion rate calculated by aggregating the conversion results in Group A (i.e., the control group) where the measure was implemented, and the conversion rate calculated by aggregating the conversion results in Group B. The certainty value ranges from 0 to 1, and a certainty value close to 1 indicates that the rule of thumb is likely to be effective in future implementations of the measure. Certainty can be calculated using various methods, but one method is as follows: The initial registration of this rule of thumb is set to 0.5, and in subsequent trials, 0.1 is added if it proves effective, and 0.1 is subtracted if it does not. This process is repeated, and the certainty increases if the rule of thumb proves effective more often than not after multiple trials.

[0027] The evaluation criteria for this rule of thumb include the date of the most recent A / B test, the number of subjects in group A, and the number of subjects in group B. The identifier for the target sample in group A is the customer ID information to which the measure was implemented.

[0028] On the other hand, for Group B, the selection method for the group corresponding to Group B is listed as "other measures," meaning that the target group to which solicitation methods other than the action measures were applied was designated as Group B. Therefore, 50 samples are randomly selected from customer users who meet these conditions, and the selected customer IDs are listed as target customers for Group B. In addition, depending on the business model, it may be possible to divide into Group A and Group B based on a choice between implementing measures or not implementing them, and in that case, Group B will be assigned to samples that meet the conditions for implementing measures but did not implement them.

[0029] Table 402 of the rules of thumb stores information about effective measures, as well as the insights that led to the creation of those measures. The data is organized by rule of thumb ID. For example, rule of thumb ID 13 in this table indicates that the target of the measure is operators of a specific age group, and the action of the measure is to assign them to the same break shift during the same time period.

[0030] Insights 1 and 2 show data trends obtained by analyzing past business performance data, and specific conditions (values) are stored for the trends in Insight 1 and Insight 2, respectively. The number of insights is not limited to two; there can be one or any other number. Furthermore, any variable itself may be stored as an insight. In addition, the content of the hypotheses derived from the insights includes hypotheses that are directly related to the measures to be implemented. This information may be automatically recorded by the system as a result of data analysis. Alternatively, the user may input each piece of information, and the system may save it. In particular, in the latter case, it is expected that the various insights obtained as a result of past data analysis will lead to the formation of groundbreaking hypotheses and the generation of measures (not merely matching or combining the conditions of the insights) based on the user's prior knowledge of the target business, and it will be possible to promote the accumulation of effective empirical rules through the verification of these hypotheses.

[0031] After updating the rule of thumb table (Environment 1) 1022 as described above, Environment 1 returns to processing S201 to generate subsequent measures. After a certain period of time has passed or when it is time to implement the next measure, data analysis and measure generation will be performed. In addition, for subsequent measure generation, measures accumulated as rules of thumb may be applied again. The series of processes from S201 to S207 in Environment 1 will continue indefinitely as long as measures for business improvement are being planned in this environment.

[0032] Next, we will explain a series of processes (S208-S214) for pre-evaluating each rule of thumb in Environment 2 using business performance data (Environment 2), assuming that the rules of thumb accumulated in Environment 1 will be applied to Environment 2. First, the latest business performance data (Environment 2) 1023 at that time is entered and stored in each storage device (S208). Next, the effectiveness of each rule of thumb accumulated in the rule of thumb table (Environment 1) 1022 in Environment 2 is pre-evaluated using business data (Environment 2) (S209).

[0033] Using the processing flow of FIG. 5, the unique pre-evaluation processing method in the present invention will be described in detail. First, from among the various rules of thumb stored in the rule-of-thumb table (environment 1) 1022, select the rule of thumb for which the effectiveness in environment 2 is to be evaluated (S501). In the rule of thumb to be pre-evaluated, search for data that satisfies the evaluation conditions used for the evaluation of the KPI effect from the business data of environment 2. Here, in the rule of thumb to be pre-evaluated, search for data that satisfies all items of the policy target conditions, policy actions, and AB test implementation time from the business performance data of environment 2. Extract the data that satisfies each of group A and group B, calculate the KPI index values of the extracted groups, and conduct an AB test (S503). The difference between the KPI index values of group A and group B is the expected value of the KPI effect amount when this rule of thumb is applied in environment 2.

[0034] Next, calculate the reliability and reproducibility of the AB test conducted in this evaluation (S504). First, regarding reliability, it is an index value indicating how reliable the conducted AB test is. As an example, generally, in an AB test, since sufficient samples are prepared for each of group A and group B and statistical superiority is evaluated using these samples, this index value can be calculated using the number of samples of each group that match the conditions and could be extracted in environment 2, and the ratio of the number of samples of group A to group B. As an example of a specific calculation method, this index value is calculated by the calculation of (number of group A / 50) × (number of group B / 50) × (number of group A / number of group B) (when the number of group A < the number of group B). By using this index, when attempting to evaluate the effect of the rule of thumb empirically evaluated by an AB test in environment 1 using the data in environment 2, it becomes possible to quantitatively calculate how accurately an AB test could be conducted from the past data of environment 2 and present it to the user.

[0035] Similarly, reproducibility is an index value obtained by calculating how well the attribute information of each target sample subjected to A / B testing matches the attribute information of the sample group in Environment 1 and the attribute information of the sample group in Environment 2. Specifically, the degree of agreement of the distribution of attribute information of the samples extracted in each group (for example, if the target sample is customer data, the distribution of attribute information such as age and gender) is calculated on both the Environment 1 and Environment 2 sides. The degree of agreement obtained for various attribute information is then statistically processed, such as by averaging, to calculate the reproducibility index value. By using this index, it is possible to quantitatively present to the user how well the evaluation status of the empirical rule demonstrated in Environment 1 (distribution of attribute information of Group A and Group B samples) can be reproduced in the pre-evaluation in Environment 2.

[0036] Furthermore, if the target policy action has never been implemented in Environment 2, there is no target sample, and both indicators will be set to 0. As described above, the effectiveness of the empirical rule of thumb to be evaluated in Environment 2 is calculated not only in terms of the estimated KPI effect size, but also in terms of its reliability and reproducibility to the prior evaluation. Through this process, for empirical rules of thumb whose reliability and reproducibility values ​​are above a certain level, it is possible to determine their effectiveness in advance by referring to their estimated effect size. The threshold values ​​for what the values ​​of both indicators should be are specified in advance by the user.

[0037] On the other hand, if the values ​​of both indicators or one of them are low, that is, if there are not enough samples that match the conditions of each group, or if the target policy action has never been implemented in Environment 2, then it is not possible to pre-evaluate the effectiveness of the relevant rule of thumb.

[0038] Therefore, in this processing flow, if the calculated reliability and reproducibility values ​​are below a certain value (S505:Yes), the consistency of the suggestive information associated with the relevant rule of thumb is checked. If the reliability and reproducibility are high (S505:No), the calculation of suggestive consistency is skipped. This reduces the amount of computation required for the preliminary evaluation.

[0039] When examining the consistency of the suggestions, the first step is to check whether or not the conditions for measuring consistency have been corrected (S506). Here, condition correction refers to artificially correcting the values ​​of the data trends, for example, when examining the consistency of suggestions in an environment where trends for various suggestions were observed in environment 1, as shown in the empirical rule table 402 in Figure 4. In this case, the focus is on whether the data in environment 2 shows the same or better trends as these data trends. Specifically, if the trend for suggestion 2 originally observed in environment 1 was an average length of service of 6.5 years, the user might consider that the data trend for this suggestion is also met in environment 2 if the average length of service is 3 years or more. This correction allows the user to perform this correction process. This enables the user to correct the conditions appropriately based on prior knowledge (general knowledge and knowledge specific to environment 2) if the data trends observed in environment 1 are not necessarily essential conditions for the validity of the corresponding empirical rule. This allows for a more accurate prior evaluation of the effectiveness of using the consistency of the suggestions of the empirical rule in environment 2.

[0040] After confirming whether or not the above-mentioned criteria for agreement have been corrected, and after appropriate corrections have been made, the degree of agreement between the trend of each suggestion in Environment 1 and the trend of the same suggestion obtained from the business data in Environment 2 is calculated (S507). As an example of a specific calculation method, the difference amount between the values ​​of each suggestion can be calculated, and statistical values ​​such as the average of these can be used. If corrections have been made, the difference amount will be calculated in accordance with the content of the corrections. If the degree of agreement of the suggestions obtained in this way is high, it can be assumed that the same effect as in Environment 1 can be expected in Environment 2, where the empirical rule is applied. Therefore, it can be used for prior evaluation of the effectiveness of measures that have not been implemented in the past.

[0041] If the preliminary evaluation of all rules of thumb has not been completed (S508: No), the process proceeds to S501, and the next rule of thumb to be evaluated is selected. The rules of thumb that have been preliminary evaluated are organized in the form of Table 601, which summarizes the preliminary evaluation results, as shown in Figure 6, and the user can refer to it as needed.

[0042] In this example, rule of thumb ID 1 represents a case where 46 samples were extracted from group A and 50 from group B in the business data of environment 2, resulting in a high reliability of 0.85 and a high reproducibility of 0.92. Based on the evaluation of the extracted groups A and B, the KPI effect size was calculated to be +25%, indicating that this rule of thumb is highly reliable and has a promising effect. Because the reliability value is high, the calculation of suggestive agreement was skipped.

[0043] In contrast, rule ID2 has a low reliability of 0.14, and this is an example of calculating the degree of agreement with the implication. The degree of agreement with the implication is high at 0.96, indicating a high probability that this rule of thumb is valid.

[0044] Heuristic ID 13 represents a measure that had never been implemented in Environment 2 before, making sample extraction impossible. Therefore, only the degree of agreement was calculated. The agreement score was a very high 0.99, indicating a high probability that this heuristic is effective.

[0045] By presenting users with a pre-evaluation table of such rules of thumb, users can refer to the pre-evaluation results from various perspectives when applying each rule of thumb to Environment 2, enabling them to efficiently carry out the process of formulating and selecting measures to be implemented in Environment 2. For example, if users want to implement rules of thumb that are certain to be effective in the actual environment, the rules of thumb may be presented in descending order of reliability. Alternatively, if users want to find rules of thumb that have not yet been implemented in Environment 2 but are highly reliable, the rules of thumb may be presented in descending order of suggestion agreement. Furthermore, the distribution of the data used in the calculation may be presented as the basis for the calculation of each pre-evaluation index value. For example, the distribution of each attribute value used in calculating the reproducibility index value may be presented.

[0046] If, after the above processing, the preliminary evaluation of all rules of thumb has been completed (S508: Yes), the preliminary evaluation process is terminated. Returning to the processing flow in Figure 2, measures to be implemented in Environment 2 are generated (S210). Similar to Environment 1, measures are generated based on insights obtained from the business performance data (Environment 2) 1023, or measures to be implemented are selected from the rules of thumb that were preliminary evaluated in S209. The measures are implemented in the same way as in Environment 1 (S211). Then, the business performance data (Environment 2) 1023 is updated (S212), and the results of the implemented measures are evaluated (S213). As a result of the evaluation, measures that were found to be effective are saved in the rules of thumb table (Environment 2) 1024, similar to Environment 1 (S214).

[0047] For subsequent processing, the latest business performance data is acquired in Environment 2 as well, and the process of generating, implementing, evaluating, and updating the rule of thumb table is repeated. The application of rule of thumb in Environment 1 may be discontinued once verification in Environment 2 is complete. Also, once the rule of thumb table has been accumulated in Environment 2, it may be applied to other different environments, similar to the case in Environment 1. In this case, if the new target environment is Environment 3, the system may be configured to perform a pre-evaluation of the application to Environment 3 by combining the accumulated rule of thumb from Environments 1 and 2. This makes it possible to pre-evaluate the effectiveness of numerous proven rule of thumb in the target environment, enabling efficient generation and selection of measures.

[0048] Next, using the example in Figure 7, we will explain the process of performing a preliminary assessment of a new environment when specific rules of thumb or suggestions exist.

[0049] The empirical rules accumulated in Environment 1, illustrated in Figure 7, show that when a customer group meeting certain conditions (in this case, data usage and age attribute values) implements measures to encourage increased data usage (in this case, providing data coupons), the KPI of the target customer group (in this case, the average payment amount of the target group) increases (Empirical Rules Table 701 in Figure 7).

[0050] The impetus for generating this measure was the discovery, through data analysis, of customer conditions where the KPI was higher compared to other condition groups (Impact 1, the condition indicated by white circle 2 in the target condition matrix 702). Based on this, the hypothesis was formulated that by implementing measures to encourage customers with conditions adjacent to that condition to transition to the condition indicated by white circle 2, they would transition to the condition with the high KPI, thus improving their KPI. Based on this hypothesis, measures were implemented for customer groups with conditions adjacent to or near condition white circle 2, and who had the potential to transition to white circle 2 through the distribution of communication coupons, and the results were evaluated to obtain the aforementioned empirical rule.

[0051] When evaluating such empirical rules in a new environment, Environment 2, if the reliability is low (for example, if the measure in question has never been implemented in Environment 2 before), as described in the explanation of the process for calculating the degree of agreement of the implications in S507 above, first check whether the trend of the accumulated implications 1 (high KPIs for the customer group with condition white circle 2) holds true for the customer group with the same conditions in Environment 2. Furthermore, check whether customers with similar attributes to customers who actually transitioned from condition white circle 5 to condition white circle 2 in Environment 1 also exist in the customer group with condition white circle 5 in Environment 2. Specifically, check the existence and number of customers in Environment 2 with similar attributes to customers with condition white circle 5 who actually transitioned to condition white circle 2 after the measure was implemented in Environment 1. Alternatively, you can compare the attribute distribution of the customer group belonging to condition white circle 5 between Environment 1 and Environment 2.

[0052] Thus, when conducting a pre-evaluation of empirical rules regarding measures that target specific conditions and encourage transitions to other desired conditions, comparing the consistency of the suggestions by examining the trends in the suggestions—specifically, the attributes of the target group under the desired conditions, the target group under the conditions that encourage transitions, and the attributes of the targets that actually transitioned—allows for a more accurate pre-evaluation of effectiveness (i.e., calculation of reliability based on that consistency). When calculating the evaluation indicators for the pre-evaluation, the above content may be included as one indicator value for the degree of suggestion consistency, or new indicators may be added and presented, such as the degree of consistency of the target group under which the measures are implemented. This allows users to understand the basis of the presented pre-evaluation more concretely.

[0053] Next, we will explain the process for accumulating empirical rules and conducting pre-evaluations for business models in which several states exist for the business subject, transitioning from the initial state (start of business) to each state, and finally reaching the final state (end of business) to obtain the final KPI.

[0054] The state transition diagram 801 shown in Figure 8 provides a simplified example of the task. The task starts in an initial state S0, and the state transitions over time or at specific timings. There are multiple types of destination states. For example, in state 1, there are two types: S1_1, where the intermediate KPI value is high, and S1_2, where the intermediate KPI value is low. Note that the criteria for distinguishing states are not limited to intermediate KPI values. In this way, several state transitions are repeated, and finally, the task ends when the task transitions to one of the final states. In this example, there are two types: Slast_1, where the final KPI value is high, and Slast_2, where the final KPI value is low. The example of state transition diagram 801 shows the actual transitions of each state in the target task, either by performing some standard task or simply by the passage of time, along with the probability of the transition. For example, the probability of transitioning from state S0 to state S1_1 is 60%, and the probability of transitioning to state S1_2 is the remaining 40%.

[0055] Figure 9 shows an example of the results of verifying the effectiveness of a specific rule of thumb in Environment 1, which is the target of such operations. In this example, the empirical experiment showed that by implementing measure 1 (white circle) from state S1_2, there is an 80% probability of transitioning to state S2_4, and further, by implementing measure 2 (white circle) in state S2_4, there is an 85% probability of transitioning to state Slast_1, which is a state with high KPIs. Such rules of thumb are accumulated as shown in the rule of thumb table 902. The aforementioned target conditions for measures and measures actions are listed, and the amount of KPI effect is compared to the case in Figure 8. Furthermore, the transition trends of each state are also accumulated. For example, the transition probability when transitioning from each state to another state, and the sample IDs of targets that actually transitioned to the intended state by implementing the measures are accumulated (in this example, for example, target samples that transitioned from state S1_2 to state S2_4 by implementing measure 1 (white circle)).

[0056] The case in which the empirical rules accumulated in Environment 1 are pre-evaluated in a different environment, Environment 2, will be explained using the state transition diagram 1001 in Figure 10. First, as described in the processes from S502 to S507, the degree of agreement with the insights that led to the implementation of A / B testing or the implementation of measures based on these empirical rules is examined from past business performance data in Environment 2.

[0057] In addition to these processes, in the case of this rule of thumb, the effectiveness of this rule of thumb can be evaluated at an intermediate stage in the implementation of measures that should be carried out consecutively (up to measure 1, indicated by the white circle in this example). Figure 1001, the state transition diagram, shows the summary of the transition from the initial state to state 1 in environment 2, and the result of implementing measure 1, indicated by the white circle, for state S1_2. We will examine whether the summary results up to this point match the trend of similar state transitions in environment 1. Specifically, we will examine the agreement between the probability of transition from state S_0 to state S1_2 and the attributes of the target of the transition, and the agreement between the attributes of the target that transitioned to state S2_4 after implementing measure 1, indicated by the white circle, for state S1_2, and the probability of that transition. If the degree of agreement of the state transitions calculated in this way is high, it is likely that the action of implementing measure 2, indicated by the white circle, for state S2_4, which is the next measure, will match the case in environment 1 with a high probability.

[0058] Therefore, once measure 1 (indicated by the white circle) is implemented, the effectiveness of subsequent measures can be predicted. In this way, when evaluating the effectiveness of empirical rules in different environments where measures are implemented successively for specific conditions, by comparing the consistency of state transition trends after the implementation of measures up to a certain point with the environment from which the data was accumulated, even for empirical rules for which there is no past performance or insight, the effectiveness can be pre-evaluated (i.e., the reliability based on that consistency can be calculated) at the stage when intermediate measures have been implemented, without having to implement all measures. This can encourage decisions such as not implementing measures for empirical rules for which no effect can be expected from subsequent measures, thereby improving the efficiency of measure implementation.

[0059] As described above, the policy evaluation device makes it possible to pre-evaluate the effectiveness of each piece of know-how accumulated in a certain environment (environment 1) using business performance data from another different environment (environment 2). In this embodiment, for simplicity, the explanation was given assuming that empirical rules accumulated in environment 1 are applied to another environment, environment 2, but the present invention is not limited to this. For example, when considering applying empirical rules accumulated in multiple environments to a certain environment, the pre-evaluation results according to the present invention can be used to select which empirical rule from which environment is most effective to apply to the target environment. This makes it possible to efficiently select the most suitable candidate empirical rule for implementation from the vast number of empirical rules accumulated in the past.

[0060] The policy evaluation device 101 may acquire information from an external system. For example, it may acquire information such as business performance data by communicating with an external device via a network.

[0061] For example, even when planning to apply empirical rules regarding carbon dioxide emission countermeasures at factory A (environment 1) to a different factory B (environment 2), the policy evaluation device 101 can perform the same processing as described above to pre-evaluate the effectiveness in a different environment. In this way, the policy evaluation device 101 can contribute to solving problems not only from an economic perspective, such as sales planning, but also from an environmental perspective.

[0062] While conversion rate was explained as an example of a KPI effect size, it is sufficient to appropriately evaluate the effectiveness of a measure, and it is not limited to conversion rate; other values ​​indicating effectiveness may also be used.

[0063] The present invention is not limited to the embodiments described above, and various design modifications can be made without departing from the spirit of the invention as described in the claims. For example, it is not necessarily limited to having all the configurations described. Furthermore, it is possible to add, delete, or replace some of the configurations of the embodiments with other configurations. [Explanation of Symbols]

[0064] 101 Policy Evaluation Device 102 Main storage 103 Control Unit 104 Secondary storage device 105 Arithmetic equipment 106 Network Interfaces 107 Input / Output Interfaces 108 Suggestion Extraction Section 109 Policy generation department 110 Policy Evaluation Department 111 Experience Accumulation Department 112 Heuristics Pre-evaluation Department 1021 Business Performance Data (Environment 1) 1022 Heuristic Table (Environment 1) 1023 Business Performance Data (Environment 2) 1024 Heuristic Table (Environment 2) 301 Business Performance Data 302 Operator Table 303 Customer Table 401 Heuristic Table 402 Heuristics Table 601 Table of Pre-Evaluation Results of Heuristics 701 Heuristic Table 702 Target Condition Matrix 801 State Transition Diagram 901 State Transition Diagram 902 Heuristic Table 1001 State Transition Diagram

Claims

1. An insight extraction unit extracts insights for business improvement from business performance data that stores the results of operations in the first environment, A policy generation unit generates measures to improve business indicators based on extracted insights, The Policy Evaluation Department evaluates the effectiveness of the measures in the first environment, The aforementioned policy evaluation unit stores policies that were effective as rules of thumb, and An empirical rule pre-evaluation unit that pre-evaluates the effectiveness of the empirical rule in a second environment different from the first environment based on business performance data in the second environment, An information output unit used for outputting information, Equipped with, The aforementioned empirical rule storage unit is, The business improvement effects of the measures in the first environment, and the data including the evaluation conditions used to assess those business improvement effects, are accumulated as empirical rules for the first environment. The aforementioned empirical rule pre-evaluation unit is: When pre-evaluating the effectiveness of the empirical rules in the first environment, data that satisfies the aforementioned evaluation conditions are extracted from the operational performance data in the second environment. Using the extracted data, predictive values ​​for business improvement effects in the second environment are calculated, and the consistency of evaluation conditions for business performance data in the first and second environments is evaluated to calculate reliability, which is a pre-evaluation value of the empirical rule. The aforementioned information output unit is: The predicted values ​​and reliability of the aforementioned business improvement effects are displayed. The aforementioned empirical rule storage unit is, We will accumulate information that provides insights into the data trends of business performance data related to the first environment, The aforementioned empirical rule pre-evaluation unit is: If the reliability value is lower than a predetermined threshold, or if it is not possible to predict the business improvement effect using the business performance data of the second environment based on the evaluation conditions, The degree of agreement between the accumulated insights trends in the first environment and the similar insights trends obtained from business performance data in the second environment is calculated, and a preliminary evaluation is conducted based on the degree of agreement. A policy evaluation device characterized by the following features.

2. A policy evaluation device according to claim 1, The aforementioned policy evaluation department, By comparing the operational indicators when the measures are implemented in the first environment with the operational indicators when the measures are not implemented in the first environment, the effectiveness of the measures in the first environment can be evaluated. The aforementioned empirical rule storage unit is, Data including evaluation conditions used when calculating business indicators under the first environment when the measures are implemented, and business indicators when the measures are not implemented under the first environment, will be accumulated as empirical rules for the first environment. A policy evaluation device characterized by the following features.

3. A policy evaluation device according to Claim 1, The aforementioned empirical rule storage unit is, In improving business operations, data on measures that encourage the target to transition to specific conditions is accumulated as empirical rules for the first environment. The aforementioned empirical rule pre-evaluation unit is: We calculate an index to determine the extent to which targets with attribute conditions that actually transitioned in the first environment due to the measures exist in the second environment, and then calculate reliability based on the consistency of the calculated values. A policy evaluation device characterized by the following features.

4. A policy evaluation device according to Claim 1, It is equipped with an information input section used for inputting information from the user, The aforementioned empirical rule storage unit is, The information regarding trends in business performance data in the first environment, which is input by the aforementioned information input unit, is stored as suggestive information. A policy evaluation device characterized by the following features.

5. A policy evaluation device according to claim 4, The aforementioned empirical rule pre-evaluation unit is: Based on the correction content that corrects the trends in the business performance data in the second environment input from the aforementioned information input unit, the degree of agreement between the trend of the suggestion in the first environment and the trend of the suggestion obtained from the business performance data in the second environment is calculated. A policy evaluation device characterized by the following features.

6. A policy evaluation device according to claim 2, The aforementioned empirical rule storage unit is, In improving business operations, data on measures that encourage the target to transition to specific conditions is accumulated as empirical rules for the first environment. The aforementioned empirical rule pre-evaluation unit is: We calculate an index to determine the extent to which targets with attribute conditions that actually transitioned in the first environment due to the measures exist in the second environment, and then calculate reliability based on the consistency of the calculated values. A policy evaluation device characterized by the following features.

7. A policy evaluation device according to Claim 2, It is equipped with an information input section used for inputting information from the user, The aforementioned empirical rule storage unit is, The information regarding trends in business performance data in the first environment, which is input by the aforementioned information input unit, is stored as suggestive information. A policy evaluation device characterized by the following features.

8. A policy evaluation device according to claim 6, It is equipped with an information input section used for inputting information from the user, The aforementioned empirical rule storage unit is, The information regarding trends in business performance data in the first environment, which is input by the aforementioned information input unit, is stored as suggestive information. A policy evaluation device characterized by the following features.

9. A policy evaluation device according to any one of claims 1 to 8, The aforementioned empirical rule storage unit is, The conditions and order of the multiple measures to be implemented, the results of each measure implemented in chronological order, and information on the probability of state transitions due to the implementation of the measures are accumulated as empirical rules for the first environment. The aforementioned empirical rule pre-evaluation unit is: The consistency between the transition probability accumulated as an empirical rule in the first environment and the transition probability in the second environment when all measures included in the empirical rule of the first environment, up to an intermediate level, are implemented in the second environment, is evaluated, and the reliability based on that consistency is calculated. A policy evaluation device characterized by the following features.

10. A method for evaluating policies using a computing device, a storage unit, and a display unit, The aforementioned computing device is From the business performance data, which stores the business performance in the first environment stored in the aforementioned memory unit, insights regarding business improvement are extracted. Based on the extracted insights, we will generate measures to improve business indicators. Evaluate the effectiveness of the measures in the first environment, Regarding measures that were deemed effective, data including the business improvement effect of the measures in the first environment and the evaluation conditions used when evaluating the said business improvement effect are stored in the memory unit as empirical rules for the first environment. Data that satisfies the aforementioned evaluation conditions is extracted from the business performance data in the second environment stored in the storage unit. Using the extracted data, predictive values ​​for business improvement effects in the second environment are calculated, and the consistency of evaluation conditions for business performance data in the first and second environments is evaluated to calculate reliability, which is a pre-evaluation value of the empirical rule. The predicted value of the aforementioned business improvement effect and the reliability are displayed on the display unit. We will accumulate information that provides insights into the data trends of business performance data related to the first environment, If the reliability value is lower than a predetermined threshold, or if it is not possible to predict the business improvement effect using the business performance data of the second environment based on the evaluation conditions, The degree of agreement between the accumulated insights trends in the first environment and the similar insights trends obtained from business performance data in the second environment is calculated, and a preliminary evaluation is conducted based on the degree of agreement. A method for evaluating policies characterized by the following features.

Images