Performance evaluation system, performance evaluation method and performance evaluation program

Abstract

To solve the problem that it becomes difficult to acquire data due to non-intervention of a control group to which a measure is not introduced.SOLUTION: A performance evaluation system has a calculation device, and a storage device, wherein the storage device holds actual performance data, the actual performance data includes an outcome index referred to so as to evaluate a result of a measure, in each period, for each of a plurality of persons, a plurality of indices to be factors of the result of the measure, and a covariant amount affecting both the result of the measure and the factor of the result of the measure, and the calculation device extracts persons corresponding to each person of an intervention group of a first group, on the basis of the covariant amount, from a plurality of persons of an intervention group of a second group and a plurality of persons of a control group of the second group, statistically analyzes the actual performance data of the persons of the intervention group of the first group, and the actual performance data of the plurality of extracted persons, and thereby evaluates the result of the measure in the first group.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to evaluation of the results of a policy. [Background technology]

[0002] In recent years, EBPM (Evidence Based Policy Making), which clarifies policy objectives and then plans policies based on evidence, has been attracting attention. In addition, with the aim of optimizing rising medical and nursing care costs, there are an increasing number of cases of pay-for-success (PFS) type contracts in the medical and nursing care fields. These efforts are premised on the existence of data to measure the results of policies and measures. International Publication No. 2018 / 225227 (Patent Document 1) is a background technology in this field. Patent Document 1 proposes a method of using the performance values ​​of similar local governments as a method of predicting the results of local government measures. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 225227 Summary of the Invention [Problem to be solved by the invention]

[0004] Japan has the world's most rapidly aging population, and is expected to continue to maintain a high aging rate in the future, raising concerns about rising social security costs, including medical and nursing care costs for the elderly. In administrative services, a high-risk approach is being promoted to introduce effective measures for elderly people who are at high risk of needing support or nursing care (high-risk group). In recent years, there have been reports of cases of performance-based contract (PFS) type businesses in which the government makes payments to private businesses according to their results, with the aim of optimizing medical and nursing care costs.

[0005] In order to introduce a PFS-type project to a specific high-risk group, it is necessary to collect information and data (evidence) that are important for measuring outcomes. The Cabinet Office has published guidelines for the quality of evidence in the "Cabinet Office Headquarters EBPM Initiative Policy for FY2018": Level 1: randomized controlled trials, Level 2a: difference-in-difference analysis, propensity score matching, instrumental variable method, Level 2b: multiple regression analysis, cohort analysis, Level 3: comparative verification, descriptive research survey, Level 4: referencing the opinions of experts, etc.

[0006] In order to obtain high-quality evidence, it is necessary to properly define an intervention group that has implemented the policy and a control group that has not implemented the policy for comparison, and to perform statistical comparison tests using data from each group before and after the implementation of the policy. However, it is usually difficult to obtain data from the control group that has not implemented the policy, since no intervention has been performed. On the other hand, the results of the policy are thought to be affected by various observable and unobservable characteristics of the municipality, such as demographics, climate and culture, and the municipality's own policies, and it is not sufficient to simply compare with similar municipalities as in existing inventions.

[0007] To address this issue, in order to properly evaluate the results of the introduced policies, we propose a performance evaluation system, method, and program that uses performance data from the intervention group in the municipality that is the subject of the policy evaluation, as well as performance data from the intervention group and control group in other municipalities that are already implementing the policy, and introduces indicators that isolate the effects of differences between municipalities, thereby analyzing the results of the policy. [Means for solving the problem]

[0008] In order to achieve at least one of the above-mentioned objects, the present invention provides an outcome evaluation system, comprising a calculation device and a storage device, the storage device storing performance data, the performance data including, for each of a plurality of individuals, an outcome indicator referenced to evaluate the result of a policy for each period, a plurality of indicators that are the cause of the result of the policy, and covariates that affect both the result of the policy and the cause of the result of the policy, the plurality of individuals including a plurality of individuals belonging to a first group to be evaluated and who participated in the policy, a plurality of individuals belonging to a second group different from the first group and who participated in the policy, and and a plurality of persons belonging to the second group who did not participate in the policy, and the calculation device extracts persons corresponding to each person who belongs to the first group and participated in the policy from the plurality of persons who belong to the second group and participated in the policy, and from the plurality of persons who belong to the second group and did not participate in the policy, based on the covariates, and evaluates the results of the policy in the first group by statistically analyzing the performance data of the plurality of persons who belong to the first group and participated in the policy and the performance data of the extracted plurality of persons. Effect of the Invention

[0009] According to one aspect of the present invention, even in a situation where a local government implements an intervention policy for residents but is unable to obtain the control group data necessary to evaluate the outcome, the outcome of the policy can be appropriately separated and evaluated by using data from other local governments. The ability to quantitatively evaluate the outcome of a policy encourages the introduction of PFS-type projects and contributes to the promotion of efficient policies that utilize the private sector. Objectives, configurations, and effects other than those described above will be clarified in the following embodiments. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram outlining a method for evaluating an intervention policy for residents implemented by a government agency in an embodiment of the present invention. [Diagram 2]FIG. 2 is an explanatory diagram of causal inference performed for evaluating an intervention measure in an embodiment of the present invention. [Diagram 3] 1 is a block diagram showing an example of the overall configuration of a system according to an embodiment of the present invention; [Figure 4] 11 is a flowchart illustrating an example of a result evaluation process according to an embodiment of the present invention. [Diagram 5] FIG. 4 is an explanatory diagram illustrating an example of a performance DB in the embodiment of the present invention. [Figure 6] FIG. 11 is an explanatory diagram showing an example of extraction of a set used for performance evaluation in an embodiment of the present invention. [Figure 7] FIG. 2 is an explanatory diagram showing an example of the relationship between coefficients obtained as a result of DID analysis in an embodiment of the present invention. [Figure 8] FIG. 13 is an explanatory diagram illustrating an example of an outcome evaluation result screen in the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention is directed to an evaluation of the results of intervention measures implemented by government agencies for residents.

[0012] FIG. 1 is a diagram for explaining an outline of a method for evaluating an intervention policy for residents implemented by a government agency in an embodiment of the present invention.

[0013] First, using Figure 1, we will explain the outline of the evaluation method for policies in City A, the target municipality that will be evaluating intervention policies for residents. Figure 1 is a table showing the data collection status for City A, the target municipality, and City B, an evaluated municipality that has already implemented the same policy and from which sufficient data has been obtained for policy evaluation. In City A, the target municipality, data before and after the intervention for the intervention group that participated in the policy has been obtained, while in City B, the evaluated municipality, in addition to data before and after the intervention for the intervention group that participated in the policy, data before and after the intervention for the control group that did not participate in the policy has also been obtained in some way.

[0014] In the present invention, residents similar to the intervention group in City A are extracted from the intervention group and control group in City B by matching, respectively, to generate a set consisting of the intervention group in City A, the intervention group in City B, and the control group in City B. A difference in difference (DID) analysis is performed on this set using a dummy variable representing the intervention group / control group, a dummy variable representing pre-intervention / post-intervention, and a dummy variable representing City A / City B. The results of the intervention policy are evaluated by removing bias in the participating group using a dummy variable representing the intervention group / control group, removing the effect of the passage of time using a dummy variable representing pre-intervention / post-intervention using a dummy variable representing City A / City B, and removing the effect due to differences in municipalities using a dummy variable representing City A / City B.

[0015] Below, we will explain the example of the evaluation of a preventive care policy that aims to reduce the number of people certified as requiring nursing care by encouraging people to go outside and participate in society.

[0016] FIG. 2 is an explanatory diagram of causal inference performed for evaluating an intervention measure in an embodiment of the present invention.

[0017] In this embodiment, as shown in Figure 2, the reduction in the number of people certified as requiring nursing care is set as an indicator of the outcome goal, and the results of the nursing care certification level 1 or higher of residents who participated in the intervention program (intervention group) and residents who did not participate (control group) are compared to evaluate the outcome of the program. In other words, the cause is whether or not they participated in the nursing care prevention program, and the result is whether or not they were certified as requiring nursing care level 1 or higher, and the program aims to evaluate the effect of participating in the program in preventing people from being certified as requiring nursing care. Here, covariates (confounding factors) that are elements that affect both the cause and the result include, for example, gender, age, medical checkup results, and medical history, and the effects are evaluated after removing the effects of these factors.

[0018] FIG. 3 is a block diagram showing an example of the overall configuration of a system according to an embodiment of the present invention.

[0019] The system of this embodiment is composed of an outcome evaluation system 100 and an administrative terminal 120. The outcome evaluation system 100 is a computer system that manages performance data of policies. The outcome evaluation system 100 is capable of communicating with the administrative terminal 120 via a network 130.

[0020] The administrative terminal 120 is a computer system that is connected to the outcome evaluation system 100 and is used to perform policy evaluation. Although one administrative terminal 120 is shown in Fig. 3, when processing data acquired from multiple municipalities, such as City A and City B described below, the administrative terminals 120 of each municipality communicate with the outcome evaluation system 100 via the network 130.

[0021] The hardware configuration of the outcome evaluation system 100 and the administrative terminal 120 is as follows. The outcome evaluation system 100 and the administrative terminal 120 each include storage devices 105 and 125, which are configured as non-volatile storage devices such as hard disk drives or built-in multimedia cards, memories 102 and 122, which are configured as volatile storage devices such as RAM, CPUs 104 and 124, which call up programs stored in the storage devices 105 and 125 into the memories 102 and 122, respectively, to control the system itself and perform various judgments, calculations, and control processes, I / O interfaces 101 and 121 for connecting to input devices and output devices, and communication devices 103 and 123, which are connected to a network 130 and communicate with other devices.

[0022] Functions implemented in the storage device 105 of the outcome evaluation system 100 include a matching extraction function 106 and an effect estimation function 107. That is, the processing of the matching extraction function 106 and the effect estimation function 107 described below is realized by calling up a program stored in the storage device 105 into the memory 102 and executing it by the CPU 104. In addition, data stored in the storage device 105 includes an outcome DB 108.

[0023] FIG. 4 is a flowchart showing an example of the result evaluation process in the embodiment of the present invention.

[0024] The outcome evaluation flow S400 of this embodiment is processed by the outcome evaluation system 100 and the administrative terminal 120. Hereinafter, the outcome evaluation flow will be explained assuming that the local government to be the subject of outcome evaluation is City A and that the local government that has already undergone outcome evaluation is City B.

[0025] The outcome evaluation flow S400 is executed upon receiving an outcome evaluation request from the administrative terminal 120 of the local government to be evaluated (City A), on the premise that the outcome evaluation system 100 holds an outcome DB 108 that stores outcome data of an evaluated local government (City B). In process S401, the administrative terminal 120 in the local government to be evaluated (City A) uploads data on the intervention group of measures implemented in City A to the outcome evaluation system 100. In uploading, personal information such as personal identification codes may be anonymized within the administrative terminal 120 as necessary.

[0026] In process S402, the matching extraction function 106 of the outcome evaluation system 100 stores the data of the intervention group of the local government to be evaluated (City A) uploaded in process S401 in the achievement DB 108, and extracts data of residents similar to the intervention group of the local government to be evaluated (City A) from the data of the evaluated local government (City B) using statistical methods such as propensity score matching. In matching, the outcome indicator is whether or not the person has received nursing care certification level 1 or higher, and the allocation variable is participation / non-participation in the intervention program, while the covariates (confounding factors) include gender, age, health checkup results, medical / pharmacy / dental receipt data, and results of health questionnaires. The propensity score is calculated by dividing the data by the number of people in the intervention group (City D), the number of people in the intervention group (City E), and the number of people in the intervention group (City F). i = 1) i (X i ) as

[0027]

number

[0028] where X iis a covariate vector consisting of covariates. In propensity score matching, data with similar propensity scores are matched and extracted.

[0029] In process S402, the matching extraction function 106 extracts a set corresponding to the intervention group of the evaluated municipality (City B) and a set corresponding to the control group by matching when extracting by propensity score matching, and creates a set to be used for outcome evaluation. An image of this set is as shown in Figure 6. An explanation of Figure 6 will be given later.

[0030] In process S403, the matching extraction function 106 of the outcome evaluation system 100 evaluates the balance of the covariates of each group extracted in the process S402, for example, using the absolute value of the standardized difference. If the absolute value of the standardized difference is greater than 0.1, the model is modified, for example, by changing the covariate vector, and the process of S402 is repeated. In modifying the model, for example, in process S404, the effect estimation function 107 of the outcome evaluation system 100 performs the following DID analysis using the data of each group extracted in the process S402.

[0031]

number

[0032] Here, Y i D is the outcome, and in this example, it is an index that is 1 when the level of care required is 1 or more, and 0 when it is less than 1. Treatment,i is a dummy variable that is equal to 1 if the patient is in the intervention group and 0 otherwise. Post,i is a dummy variable that is equal to 1 if the intervention is completed and 0 otherwise. A,i e is a dummy variable that is 1 if the city is A and 0 otherwise. i is the residual.

[0033] β3, obtained as a result of this DID analysis, represents the effect obtained by the intervention measures regardless of whether it is City A or City B, and β4 represents the effect obtained by the intervention measures in City A. The relationship between the coefficients is shown in Figure 7. An explanation of Figure 7 will be given later.

[0034] In process S405, the administrative terminal 120 receives the outcome evaluation result from the outcome evaluation system 100, and displays, for example, the outcome evaluation result as shown in Fig. 8. Fig. 8 will be described later.

[0035] FIG. 5 is an explanatory diagram showing an example of the result DB 108 in the embodiment of the present invention.

[0036] The performance DB108 holds time-series data on outcomes, covariates, allocation variables, and municipalities of residence for individuals represented by specific IDs, etc. In this example, the outcome is the care level (1: care level 1 or higher, 0: care level less than 1), the covariates are health check, medical receipt, and questionnaire data, the allocation variables are the intervention group (1: intervention group, 0: control group), and the municipality of residence is the ward, city, town, or village (1: City A, 0: City B).

[0037] Specifically, the performance DB 108 shown in FIG. 5 includes a plurality of records each consisting of a serial number 501 , an ID 502 , a year 503 , a care need category 504 , an intervention group 505 , before and after intervention 506 , a covariate 507 , and a city 508 .

[0038] Each record in the performance DB 108 (i.e., each row in the table shown in FIG. 5) includes information on an individual identified by an ID 502 and a year identified by a year 503. A serial number 501 is a number that uniquely identifies each record, and corresponds to i in the above formulas (1) and (2). A care need category 504 is an outcome.

[0039] The intervention group 505 is a variable indicating whether the person in each record belongs to the intervention group. Before and after intervention 506 is a variable indicating whether the year of the record is before or after the implementation of the intervention measure in the municipality in which the person in the record lives, regardless of whether the person in the record belongs to the intervention group (1: after implementation, 0: before implementation).

[0040] Covariates 507 are data on medical checkups, medical receipts, questionnaires, etc. obtained for each record person in each year, and generally include many items, although they are omitted in Fig. 5. City, ward, town, village 508 indicates whether the person in each record lived in City A in each year.

[0041] For example, the records of i=1 and i=2 in the performance DB108 shown in Figure 5 indicate that a person identified by "A01" (hereinafter, this will also be simply referred to as "Person A01". Persons identified by other identifiers will be written in a similar manner) resides in City A, participates in a program that began in 2022, and has not been certified as requiring nursing care level 1 or higher in either 2021 or 2022. In this case, the above formula (2) becomes the following formulas (3) and (4), respectively.

[0042] 0=β0+β1×1+β2×0+β3×1×0+β4×1×0×1+β5×1+e1(3)

[0043] 0=β0+β1×1+β2×1+β3×1×1+β4×1×1×1+β5×1+e2(4)

[0044] Furthermore, the records of i=101 and i=102 in the performance DB 108 shown in FIG. 5 indicate that person B01 resides in City B, does not participate in the program that began in 2022, and was not certified as needing nursing care level 1 or higher in 2021, but was certified as needing nursing care level 1 or higher in 2022. In this case, the above formula (2) becomes the following formulas (5) and (6), respectively.

[0045] 0=β0+β1×0+β2×0+β3×0×0+β4×0×0×0+β5×0+e 101 (5)

[0046] 1=β0+β1×0+β2×1+β3×0×1+β4×0×1×0+β5×0+e 102 (6)

[0047] Similarly, β0 to β5 can be obtained by substituting the data for each individual for each year recorded in the performance DB 108 into the outcome and dummy variables in equation (2) and performing a DID analysis.

[0048] FIG. 6 is an explanatory diagram showing an example of extraction of a set used for performance evaluation in the embodiment of the present invention.

[0049] In the example of FIG. 6, the probability (i.e., propensity score) calculated by applying the covariate 507 of person A01 in the intervention group in City A to formula (1) is 0.951. In this case, people with propensity scores close to 0.951 are extracted from the intervention group and the control group in City B. In the example of FIG. 6, person b01 with a propensity score of 0.948 is extracted from the intervention group in City B, and person B01 with a propensity score of 0.952 is extracted from the control group in City B. The fact that the propensity scores of these people are close indicates that the contents of their medical checkup data, medical receipts, questionnaires, etc. are similar. Similarly, for other people in the intervention group in City A, such as person A02, people in the intervention group and the control group in City B with similar propensity scores are extracted.

[0050] By extracting individuals with similar propensity scores in this manner, the impact of differences in covariates on the evaluation of the outcomes of the intervention can be reduced, enabling the outcomes of the intervention to be evaluated with high accuracy.

[0051] FIG. 7 is an explanatory diagram showing an example of the relationship between the coefficients obtained as a result of the DID analysis in the embodiment of the present invention.

[0052] According to equation (2), the pre-intervention outcomes of the intervention group in City A are associated with β0, β1, and β5 (in other words, not associated with β2, β3, and β4). In contrast, the post-intervention outcomes of the intervention group in City A are associated with β0, β1, β2, β3, β4, and β5. Meanwhile, the pre-intervention outcomes of the control group in City A are associated with β0 and β5. In contrast, the post-intervention outcomes of the control group in City A are associated with β0, β2, and β5.

[0053] Similarly, pre-intervention outcomes for the intervention group in City B are associated with β0 and β1. Post-intervention outcomes for the intervention group in City B are associated with β0, β1, and β2. Pre-intervention outcomes for the control group in City B are associated with β0. Post-intervention outcomes for the control group in City B are associated with β0 and β2.

[0054] The relationship between the outcomes and each coefficient above indicates that β3 represents the effect of the intervention measures regardless of place of residence (i.e., whether it is City A or City B), and β4 represents the effect of the intervention measures in City A. Additionally, β2 represents the magnitude of the effect that the passage of time simply has on the outcome, regardless of the intervention measures and regardless of whether one lives in City A or City B. Additionally, β5 represents the magnitude of the effect that living in City A has on the outcome, regardless of the intervention measures and regardless of the passage of time.

[0055] This allows us to separate effects that are not based on the intervention (for example, effects due only to the passage of time and effects due only to the place of residence) and accurately evaluate the results of the intervention.

[0056] FIG. 8 is an explanatory diagram showing an example of the outcome evaluation result screen in the embodiment of the present invention.

[0057] The outcome evaluation result screen 800 shown in Fig. 8 includes an intervention group attribute display section 801 and an intervention measure result display section 802. The intervention group attribute display section 801 displays the attributes of the people in the intervention group in City A, which is the local government to be evaluated. In the example of Fig. 8, the number of people in the intervention group by age and gender is displayed. These attributes can be obtained from the covariates 507.

[0058] The intervention measure result display section 802 displays, as a result of evaluating the success of the intervention measures, the intervention effect common to all cities, wards, towns, and villages (i.e., regardless of whether the place of residence is City A or City B), the intervention effect specific to residents of City A, and the total intervention effect. In the example of FIG. 8, the suppression effect on the nursing care certification rate is displayed as each intervention effect. The intervention effect common to all cities, wards, towns, and villages is calculated based on β3 in formula (2), and the intervention effect specific to City A is calculated based on β4 in formula (2), and the total intervention effect is the sum of these values.

[0059] The embodiment of the present invention described above makes it possible to evaluate the results of an intervention policy implemented in City A, the target municipality. Furthermore, if there are multiple municipalities that have already been evaluated, it is possible to perform outcome evaluation by matching and extracting intervention and control groups from the multiple municipalities. By using data from multiple municipalities, it is expected that factors that are common to City A and City B but not included in the other municipalities can be isolated and eliminated.

[0060] For example, a dummy variable D is 1 if the person lives in city B and 0 otherwise. B,i A DID analysis can be carried out by adding a term including A,i =1, D B,i = 0, and for residents of city B, D A,i =0, D B,i = 1, and for residents of city C, D A,i =0, D B,i =0.

[0061] Furthermore, the system according to the embodiment of the present invention may be configured as follows.

[0062] (1) An outcome evaluation system, comprising a calculation device (e.g., CPU 104) and a storage device (e.g., memory 102 and storage device 105), the storage device holds performance data (e.g., performance DB), the performance data includes, for each of a plurality of persons, an outcome indicator (e.g., care need category 504) to be referenced for evaluating the result of a policy for each period, a plurality of indicators that are the cause of the result of the policy (e.g., intervention group 505, before and after intervention 506, city 508), and a covariate (e.g., covariate 507) that affects both the result of the policy and the cause of the result of the policy, the plurality of persons belonging to a first group to be evaluated and a plurality of persons who participated in the policy (e.g., persons in the intervention group in City A), belonging to a second group different from the first group, The first group includes a plurality of individuals who belong to the first group and participated in the policy (e.g., individuals in an intervention group in city B), and a plurality of individuals who belong to the second group and did not participate in the policy (e.g., individuals in a control group in city B), and the calculation device extracts individuals corresponding to each individual who belongs to the first group and participated in the policy from the plurality of individuals who belong to the second group and participated in the policy, and the plurality of individuals who belong to the second group and did not participate in the policy, based on the covariates (e.g., process S402), and evaluates the results of the policy in the first group by statistically analyzing the performance data of the plurality of individuals who belong to the first group and participated in the policy, and the performance data of the extracted plurality of individuals (e.g., process S404).

[0063] This means that, for example, even in situations where a local government implements an intervention policy for residents but is unable to obtain the control group data necessary to evaluate the results, it is possible to properly isolate and evaluate the results of the policy by using data from other local governments.

[0064] (2) In the performance evaluation system described in (1) above, the calculation device extracts individuals who belong to the first group and have similar tendencies in the covariates to each individual who participated in the program from among a plurality of individuals who belong to the second group and participated in the program, and from a plurality of individuals who belong to the second group and did not participate in the program.

[0065] This reduces the impact of differences in covariates on the evaluation of the results of the policy, enabling the results of the policy to be evaluated with greater accuracy.

[0066] (3) In the outcome evaluation system described in (2) above, the multiple indicators that are the cause of the outcome of the policy include an indicator indicating whether each of the persons participated in the policy (e.g., intervention group 505), an indicator indicating whether each of the persons belongs to the first group (e.g., city / ward / town / village 508), and an indicator indicating whether each of the periods is after the implementation of the policy (e.g., before / after intervention 506), and the calculation device calculates a variable (e.g., D Treatment,i ), a variable (e.g., D A,i ), and a variable (e.g., D Post,i ) and performing a difference-in-difference analysis using a formula for calculating the outcome index, the difference-in-difference analysis evaluates at least one of the results of the measures that are common regardless of the group to which the person belongs (e.g., based on β3) and the results of the measures that are specific to the person belonging to the first group (e.g., based on β4).

[0067] This makes it possible to separate effects that are not based on the policy (for example, effects due only to the passage of time and effects due only to the place of residence) and accurately evaluate the results of the policy.

[0068] (4) The performance evaluation system described in (3) above, wherein the first group is a group of people living in a first municipality (e.g., City A), the second group is a group of people living in a second municipality (e.g., City B), the measures are measures to prevent the people from needing nursing care, the outcome indicator is an indicator indicating whether the people have reached a state where they need nursing care (e.g., nursing care certification level 1 or higher), and the covariates include at least one of the people's health check data, prescription data, and questionnaire data.

[0069] This will enable accurate evaluation of the effectiveness of interventions to reduce the risk of needing nursing care.

[0070] (5) The performance evaluation system described in (3) above, wherein the plurality of individuals further includes a plurality of individuals belonging to a third group different from both the first group and the second group and who participated in the policy, and a plurality of individuals belonging to the third group and who did not participate in the policy, and the calculation device extracts individuals having similar tendencies of the covariates to each of the individuals belonging to the first group and who participated in the policy from the plurality of individuals belonging to the third group and who participated in the policy, and from the plurality of individuals belonging to the third group and who did not participate in the policy, respectively, and evaluates at least one of the results of the policy that are common regardless of the group to which the individuals belong, the results of the policy that are specific to individuals belonging to the first group, and the results of the policy that are specific to individuals belonging to the first group and individuals belonging to the second group.

[0071] This will make it possible to isolate and eliminate factors that are common to City A and City B but not included in other municipalities such as City C.

[0072] The present invention is not limited to the above-mentioned embodiment, but includes various modifications. For example, the above-mentioned embodiment has been described in detail for better understanding of the present invention, and is not necessarily limited to those including all of the configurations described. Also, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Also, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0073] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function can be stored in a storage device such as a non-volatile semiconductor memory, a hard disk drive, or an SSD (Solid State Drive), or in a computer-readable non-transitory data storage medium such as an IC card, an SD card, or a DVD.

[0074] In addition, the control lines and information lines are shown as those considered necessary for the explanation, and not all control lines and information lines in the product are shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0075] 100 Performance Evaluation System 101, 121 I / O Interface 102, 122 memory 103, 123 Communication equipment 104, 124 CPU (computing unit) 105, 125 Storage device 106 Matching extraction function 107 Effect Estimation Function 108 Achievement DB

Claims

1. 1. A performance evaluation system, comprising: A computing device and a storage device, The storage device holds performance data, The performance data includes, for each of a plurality of persons, an outcome indicator referred to for evaluating the result of a policy, a plurality of indicators that are the cause of the result of the policy, and a covariate that affects both the result of the policy and the cause of the result of the policy, The plurality of individuals include a plurality of individuals who belong to a first group that is the subject of evaluation and who participated in the campaign, a plurality of individuals who belong to a second group different from the first group and who participated in the campaign, and a plurality of individuals who belong to the second group and who did not participate in the campaign; The computing device includes: extracting, from a plurality of individuals who belong to the second group and participated in the campaign, and a plurality of individuals who belong to the second group and did not participate in the campaign, individuals corresponding to each of the individuals who belong to the first group and participated in the campaign, based on the covariates; A performance evaluation system that evaluates the results of the policy in the first group by statistically analyzing the performance data of a plurality of individuals who belong to the first group and participated in the policy, and the performance data of the extracted plurality of individuals.

2. The outcome evaluation system according to claim 1, The computing device is an outcome evaluation system characterized in that it extracts individuals who belong to the first group and have similar tendencies in the covariates to each individual who participated in the policy from a plurality of individuals who belong to the second group and participated in the policy, and from a plurality of individuals who belong to the second group and did not participate in the policy.

3. The outcome evaluation system according to claim 2, The multiple indicators that are the cause of the outcome of the policy include an indicator indicating whether or not each of the persons participated in the policy, an indicator indicating whether or not each of the persons belongs to the first group, and an indicator indicating whether or not each of the periods occurred after the implementation of the policy; The calculation device includes a variable into which an indicator indicating whether each of the persons participated in the policy, a variable into which an indicator indicating whether each of the persons belongs to the first group, and a variable into which an indicator indicating whether each of the periods is after the implementation of the policy is assigned, based on the performance data of a plurality of persons who belong to the first group and participated in the policy, and the performance data of the extracted plurality of persons, and by performing a difference-in-difference analysis using a formula for calculating the outcome index, the system evaluates at least one of the results of the policy that are common regardless of the group to which a person belongs and the results of the policy that are specific to persons belonging to the first group.

4. The outcome evaluation system according to claim 3, the first population is a population of persons residing in a first municipality; the second population is a population of persons residing in a second municipality; The measure is a measure for preventing the person from becoming in need of care, The outcome index is an index indicating whether or not the person has become in need of care, An outcome evaluation system characterized in that the covariates include at least one of the person's health check data, prescription data, and questionnaire data.

5. The outcome evaluation system according to claim 3, the plurality of persons further includes a plurality of persons who belong to a third group different from both the first group and the second group and who participated in the campaign, and a plurality of persons who belong to the third group and who did not participate in the campaign; The computing device includes: Extracting individuals who belong to the first group and have similar tendencies in the covariates to those of the individuals who participated in the program from among the multiple individuals who belong to the third group and participated in the program, and from among the multiple individuals who belong to the third group and did not participate in the program; An outcome evaluation system characterized by evaluating at least one of the results of the measures that are common regardless of the group to which a person belongs, the results of the measures that are specific to individuals belonging to the first group, and the results of the measures that are specific to individuals belonging to the first group and individuals belonging to the second group.

6. A performance evaluation method using a performance evaluation system, comprising: The outcome evaluation system includes a computing device and a storage device, The storage device holds performance data, The performance data includes, for each of a plurality of persons, an outcome indicator referred to for evaluating the result of a policy, a plurality of indicators that are the cause of the result of the policy, and a covariate that affects both the result of the policy and the cause of the result of the policy, The plurality of individuals include a plurality of individuals who belong to a first group that is the subject of evaluation and who participated in the campaign, a plurality of individuals who belong to a second group different from the first group and who participated in the campaign, and a plurality of individuals who belong to the second group and who did not participate in the campaign; The outcome evaluation method includes: a first step in which the arithmetic device extracts, from a plurality of individuals who belong to the second group and participated in the campaign, and a plurality of individuals who belong to the second group and did not participate in the campaign, individuals corresponding to each of the individuals who belong to the first group and participated in the campaign, based on the covariates; a second step in which the calculation device evaluates the results of the policy in the first group by statistically analyzing the performance data of a plurality of individuals who belong to the first group and participated in the policy, and the performance data of the extracted plurality of individuals.

7. The outcome evaluation method according to claim 6, An outcome evaluation method characterized in that in the first step, the calculation device extracts individuals who belong to the first group and have similar tendencies in the covariates to each individual who participated in the policy from a plurality of individuals who belong to the second group and participated in the policy, and from a plurality of individuals who belong to the second group and did not participate in the policy.

8. The outcome evaluation method according to claim 7, The multiple indicators that are the cause of the outcome of the policy include an indicator indicating whether or not each of the persons participated in the policy, an indicator indicating whether or not each of the persons belongs to the first group, and an indicator indicating whether or not each of the periods occurred after the implementation of the policy; The second step includes a variable into which an indicator indicating whether each of the persons participated in the policy, a variable into which an indicator indicating whether each of the persons belongs to the first group, and a variable into which an indicator indicating whether each of the periods is after the implementation of the policy is assigned, based on the performance data of a plurality of persons who belong to the first group and participated in the policy, and the performance data of the extracted plurality of persons, and the calculation device performs a difference-in-difference analysis using a formula for calculating the outcome index, thereby evaluating at least one of the results of the policy that are common regardless of the group to which a person belongs and the results of the policy that are specific to a person belonging to the first group.

9. The outcome evaluation method according to claim 8, the first population is a population of persons residing in a first municipality; the second population is a population of persons residing in a second municipality; The measure is a measure for preventing the person from becoming in need of care, The outcome index is an index indicating whether or not the person has become in need of care, An outcome evaluation method, characterized in that the covariates include at least one of the person's health check data, prescription data, and questionnaire data.

10. The outcome evaluation method according to claim 8, the plurality of persons further includes a plurality of persons who belong to a third group different from both the first group and the second group and who participated in the campaign, and a plurality of persons who belong to the third group and who did not participate in the campaign; In the first step, the calculation device extracts individuals who belong to the first group and have a similar tendency in the covariate to each individual who participated in the campaign from among a plurality of individuals who belong to the third group and have participated in the campaign, and a plurality of individuals who belong to the third group and have not participated in the campaign, A method for evaluating results, characterized in that in the second step, the calculation device evaluates at least one of the results of the measures that are common regardless of the group to which a person belongs, the results of the measures that are specific to a person belonging to the first group, and the results of the measures that are specific to a person belonging to the first group and a person belonging to the second group.

11. A performance evaluation program for controlling a performance evaluation system, The outcome evaluation system includes a computing device and a storage device, The storage device holds performance data, The performance data includes, for each of a plurality of persons, an outcome indicator referred to for evaluating the result of a policy, a plurality of indicators that are the cause of the result of the policy, and a covariate that affects both the result of the policy and the cause of the result of the policy, The plurality of individuals include a plurality of individuals who belong to a first group that is the subject of evaluation and who participated in the campaign, a plurality of individuals who belong to a second group different from the first group and who participated in the campaign, and a plurality of individuals who belong to the second group and who did not participate in the campaign; The outcome evaluation program is a first step of extracting, from a plurality of individuals who belong to the second group and participated in the campaign, and a plurality of individuals who belong to the second group and did not participate in the campaign, individuals corresponding to each of the individuals who belong to the first group and participated in the campaign, based on the covariates; a second step of evaluating the results of the policy in the first group by statistically analyzing the performance data of a plurality of individuals who belong to the first group and participated in the policy, and the performance data of the extracted plurality of individuals.

Images