Outcome evaluation device and outcome evaluation method

Abstract

To provide a technology to collect and analyze data before and after introduction of measures to appropriately evaluate the effectiveness of the measures.SOLUTION: An achievement evaluation apparatus that comprises a processor and a memory and evaluates measures provided by the government, comprises: user information in which the state of a user before and after implementation of each measure is stored in advance as an evaluation index for each of the measures; intervention measure information specifying in advance a measure to be evaluated, a target outcome for evaluating the measure, an evaluation target period, and an evaluation process; and an intervention measure management unit for receiving the intervention measure information and evaluating the effectiveness of the measure to be evaluated using the evaluation index of the user information for the measure to be evaluated. The intervention measure management unit acquires the user information of the user who has implemented at least one of the measures provided during the evaluation target period specified in the intervention measure information, and evaluates the effectiveness of the measure to be evaluated through the evaluation process based on the evaluation index corresponding to the target outcome.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for evaluating the results of a policy. [Background technology]

[0002] In recent years, evidence-based policymaking (EBPM), which involves clarifying policy objectives and formulating policies based on evidence, has been gaining attention. Furthermore, with the aim of optimizing rising medical and nursing care costs, there has been an increase in the number of cases of pay-for-success (PFS) contracts in the medical and nursing care fields.

[0003] These efforts are premised on the availability of data for measuring the effects of policies and measures. Patent Document 1 is known as background technology in this field. Patent Document 1 proposes a system that uses basic data on elderly people, data on nursing care insurance needs, medical insurance data, and data on local measures to output quantitative analysis reports and qualitative reports for each business unit, and supports the confirmation of the effects of care plans for community-based integrated care and the consideration of policies for reviewing the next plan. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-87239 Summary of the Invention [Problem to be solved by the invention]

[0005] Japan has the world's fastest aging population and is expected to maintain a high aging rate in the future, raising concerns about rising social security costs, including medical and nursing care expenses for the elderly. In administrative services, a high-risk approach is being promoted, introducing effective measures for elderly people at high risk of needing assistance or nursing care (high-risk groups). In addition, in recent years, there have been reported cases of performance-based contract (PFS) projects, in which the government makes payments based on the results of private projects, with the aim of optimizing medical and nursing care costs.

[0006] In order to introduce a PFS-type project for a specific high-risk group, it is necessary to collect information and data (evidence) that are important for measuring effectiveness. Regarding the level of evidence, the Cabinet Office has published guidelines in its "Cabinet Office Headquarters EBPM Initiative Policy for FY2018" as follows: Level 1: randomized controlled trials; Level 2a: difference-in-difference analysis, propensity score matching, instrumental variable method; Level 2b: multiple regression analysis, cohort analysis; Level 3: comparative verification, descriptive research survey; and Level 4: reference to expert opinions.

[0007] To obtain high-quality evidence, it is necessary to properly define an intervention group that has implemented the intervention measures and a control group that has not implemented the measures for comparison, and to conduct statistical comparison tests using data from each group before and after the implementation of the measures.

[0008] However, in previous examples, it was difficult to obtain data from a control group that had not received the intervention. Furthermore, existing data on medical care and nursing care consisted of data on people who underwent annual health checkups, or data on occasional events such as certification of need for assistance or nursing care, or visits to medical institutions, and this data is often insufficient to evaluate the effects of policies before and after their introduction.

[0009] To address the above issues, the objective is to provide a technology that collects and analyzes data before and after the implementation of a policy to appropriately evaluate the effectiveness of the policy's implementation. [Means for solving the problem]

[0010] The present invention is an outcome evaluation device having a processor and memory for evaluating policies provided by the government, and includes user information in which the user's status before and after the implementation of each of a plurality of policies is pre-stored as an evaluation index, intervention policy information in which the policy to be evaluated, a target outcome for evaluating the policy, a target evaluation period, and an evaluation process are pre-specified, and an intervention policy management unit that receives the intervention policy information and evaluates the effectiveness of the policy to be evaluated using the evaluation index of the user information for the policy to be evaluated, wherein the intervention policy management unit obtains user information of users who implemented at least one of the policies provided during the target evaluation period specified in the intervention policy information, and evaluates the effectiveness of the policy to be evaluated using the evaluation index corresponding to the target outcome through the evaluation process. [Effects of the Invention]

[0011] This invention allows governments to simultaneously introduce multiple policies for residents and quantitatively evaluate the effectiveness of each policy through statistical analysis. The ability to quantitatively evaluate the effectiveness of policies will encourage the introduction of PFS-type projects and contribute to the promotion of efficient intervention policies that utilize the private sector.

[0012] The details of at least one implementation of the subject matter disclosed herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the disclosed subject matter will become apparent from the following disclosure, drawings, and claims. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 2 is a diagram showing a pattern of data used in the evaluation method implemented by the evaluation result system in an embodiment of the present invention. [Figure 2] FIG. 1 is a Venn diagram showing an image of an ideal data set to be collected in a randomized controlled experiment according to an embodiment of the present invention. [Figure 3] FIG. 1 is a Venn diagram showing a desirable state for performing propensity score matching and DID analysis in an embodiment of the present invention. [Figure 4A]FIG. 10 is a Venn diagram illustrating a state in which propensity score matching and DID analysis cannot be performed in an embodiment of the present invention. [Figure 4B] FIG. 10 is a Venn diagram illustrating a state in which propensity score matching and DID analysis cannot be performed in an embodiment of the present invention. [Figure 5] 1 is a block diagram showing an example of the configuration of a computer system including a result evaluation device, a user client terminal, and an administrative terminal in an embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating an example of a procedure for registering user information according to an embodiment of the present invention. [Figure 7] 10 is a flowchart illustrating an example of a procedure for setting an intervention measure according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram showing an example of an intervention measure management screen in an embodiment of the present invention. [Figure 9] FIG. 10 is a diagram showing an example of an intervention measure details screen in the embodiment of the present invention. [Figure 10] 1 is a flowchart showing an example of a procedure for implementing an intervention measure according to an embodiment of the present invention. [Figure 11] 1 is a flowchart showing an example of a procedure for evaluating an intervention measure in an embodiment of the present invention. [Figure 12] FIG. 10 is a diagram illustrating an example of a user table according to an embodiment of the present invention. [Figure 13] FIG. 10 is a diagram showing an example of an intervention measure utilization table in an embodiment of the present invention. [Figure 14] FIG. 10 is a diagram showing an example of an outcome table in an embodiment of the present invention. [Figure 15] FIG. 10 is a diagram showing an example of an intervention policy table in the embodiment of the present invention. [Figure 16] FIG. 1 is a diagram showing an example of a dataset used for statistical analysis in an example of the present invention. [Figure 17] FIG. 10 is a diagram showing an example of timing of data acquisition and whether or not an intervention measure is implemented in an embodiment of the present invention. [Figure 18] FIG. 10 is a diagram showing an example of a basic checklist in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] An embodiment of the present invention will be described in detail below with reference to the accompanying drawings. This embodiment is directed to the evaluation of the results of a policy implemented by a government agency that includes interventions for residents (hereinafter referred to as "intervention policy").

[0015] First, using Figures 1 to 4, we will provide an overview of the evaluation method for intervention measure P003 when intervention measures P001, P002, and P003 are implemented simultaneously.

[0016] Figure 1 shows a pattern 20 of intervention data. Pattern 20 shows a combination of whether or not intervention measures P001 (201), P002 (202), and P003 (203) were implemented (1 (implemented), 0 (not implemented)) for each subject (or subject group) # (200).

[0017] Subject #200 stores an identifier for identifying the subject (or subject group) who will receive an intervention measure. Intervention measure P001 (201) stores "1" (implemented) in the record of subject #200 who has implemented the intervention measure, and stores "0" (not implemented) in the record of subject #200 who has not implemented the intervention measure. Similarly, intervention measure P002 (202) and intervention measure P003 (203) store whether the intervention measure has been implemented or not.

[0018] The outcome evaluation device 100 can evaluate before and after the implementation of the intervention measure P003 (203) by comparing the patterns of the subjects #201 = #1 and #4. For example, as shown in Figure 2, a randomized controlled experiment randomly divides participants 212 who implemented the intervention measure P003, which has the highest level of quality of evidence, from participants 211 who did not implement the intervention measure P003, and collects and compares the data from each group for analysis.

[0019] In such analyses, to avoid the influence of other factors, it is common to match the conditions of the two groups except for the implementation of a specific intervention (for example, by standardizing all other intervention measures as not implemented).

[0020] However, from the perspective of fairness, it is undesirable to predetermine who will not receive an intervention measure, so such randomized controlled trials are often conducted in academic research and are not compatible with government services.

[0021] Furthermore, those who have not undergone intervention have not received any intervention and usually have no contact with those who implement the intervention. In order to collect data from such people, it is necessary to set them as the control group (non-intervention group) in academic research such as the one mentioned above and intentionally collect data from them.

[0022] In this example, multiple interventions (P001-P003) are implemented simultaneously, and data is collected under conditions where intervention subjects (users) randomly choose whether to implement or not implement the interventions. Using data from residents who implemented one or more of the interventions (#2-#8 in Figure 1), well-known statistical analyses, such as propensity score matching and DID analysis (Difference-in-Difference), which provide high-quality evidence, are performed to evaluate each intervention (here, intervention P003).

[0023] In this example, it is assumed that whether or not to participate in an intervention measure is left to the free will of the residents, and therefore there are cases where propensity score matching or DID analysis cannot be performed as a result of implementing an intervention measure.

[0024] As shown in Figure 3, when data for each pattern exists, this is desirable for propensity score matching and DID analysis. On the other hand, when there are no people who implemented multiple intervention measures, as shown in Figure 4A, or when the implementers of multiple intervention measures are completely identical, as shown in Figure 4B, it is not possible to calculate the effects using propensity score matching or DID analysis. In such cases, a simple before-and-after comparison is adopted as the next best evaluation method.

[0025] Below, we will explain how we support the planning of intervention measures to collect data for each of these patterns and how we analyze the collected data.

[0026] 5 is a block diagram showing an example of the configuration of the outcome evaluation system in this embodiment. The outcome evaluation system is a computer system in which an outcome evaluation device 100, a user client terminal 120, an administrative terminal 130, and an enterprise server 300 are connected via a network 140.

[0027] The outcome evaluation device 100 is a computer that manages user information (user table 1200, intervention measure usage table 1300), outcome information (outcome table 1400), and intervention measure information (intervention measure table 1500).

[0028] The outcome evaluation device 100 is capable of communicating with a user client terminal 120 and an administrative terminal 130 via a network 140. The user client terminal 120 is a computer connected to the outcome evaluation device 100 and used by users to manage user information. The administrative terminal 130 is a computer connected to the outcome evaluation device 100 and used to manage outcome information and intervention measure information.

[0029] The hardware configurations of the outcome evaluation device 100, user client terminal 120, and administrative terminal 130 are as follows: The outcome evaluation device 100, user client terminal 120, and administrative terminal 130 include storage devices 105, 125, and 135, which are nonvolatile storage devices such as hard disk drives and built-in multimedia cards; memories 102, 122, and 132, which are volatile storage devices such as RAM; CPUs 104, 124, and 134, which call programs stored in the storage devices 105, 125, and 135 into the memories 102, 122, and 132 to perform overall control of the system itself and perform various judgments, calculations, and control processing; I / O interfaces 101, 121, and 131 for connecting to input and output devices; and communication devices 103, 123, and 133, which are connected to a network 140 and communicate with other devices.

[0030] The functions implemented in the storage device 105 of the outcome evaluation device 100 include a user management function 106 and an intervention measure management function 108. The user management function 106 is a functional unit (user management unit) configured by a user management program. The intervention measure management function 108 is also a functional unit (intervention measure management unit) configured by an intervention measure management program. Each program is loaded into the memory 102 and then executed by the CPU 104.

[0031] The CPU 104 operates as a functional unit that provides a predetermined function by processing in accordance with the program of each functional unit. For example, the CPU 104 provides a user management function 106 by processing in accordance with a user management program. The same applies to other programs. Furthermore, the CPU 104 also operates as a functional unit that provides each function of the multiple processes executed by each program. A computer and a computer system are devices and systems that include these functional units.

[0032] The data stored in the storage device 105 include a user DB (database, hereinafter the same) 107 and an intervention measure DB 109. The user DB 107 includes a user table 1200 and an intervention measure usage table 1300. The intervention measure database 109 includes an outcome table 1400 and an intervention measure table 1500.

[0033] The business operator server 300 is a computer having the same hardware as the outcome evaluation device 100 and the like, and provides an intervention measure 310 to users on commission from the government.

[0034] 6 shows an example of the user information registration process S600 in this embodiment. The user information registration process S600 in this embodiment is processed by the user client terminal 120 and the user management function 106 of the outcome evaluation device 100.

[0035] This process is performed when a user inputs his / her own information, on the assumption that the user has already been registered in the user DB 107 of the outcome evaluation device 100. The user information registration process will be described in detail below.

[0036] In process S601, the user client terminal 120 transmits a user login request to the outcome evaluation device 100 via the user management function 106 provided by the outcome evaluation device 100. At this time, the information transmitted from the user client terminal 120 includes, for example, a user ID assigned to each user and a user password preset for each user.

[0037] In process S602, the outcome evaluation device 100 performs login authentication based on the login request received from the user client terminal 120. The login authentication is performed, for example, by comparing the pair of user ID and user password received from the user client terminal 120 with the pair of user ID and user password registered in advance in the outcome evaluation device 100.

[0038] If the user login is successful, the outcome evaluation device 100 proceeds to step S603, and if the login is unsuccessful, it notifies the user client terminal 120 of the login failure and requests the user to enter the user ID and password again.

[0039] In process S603, the user management function 106 of the outcome evaluation device 100 outputs a screen for inputting user information to the user client terminal 120 of the user who has successfully logged in and been authenticated.

[0040] In step S604, the user client terminal 120 accepts input of user information from the user. The user inputs basic information such as name and address, as well as information such as family structure and occupation.

[0041] The user client terminal 120 that receives input from the user transmits the input content to the outcome evaluation device 100. Here, it is necessary to acquire as many items as possible that may affect the effectiveness of the intervention measures that the outcome evaluation device 100 wants to evaluate. In addition, for example, data on residents held by the government (national health insurance medical receipt data, health checkup data for the elderly, certification data on need for support / nursing care, etc.) may also be linked as much as possible.

[0042] In process S605, the outcome evaluation device 100 stores the user information (user attribute information, etc.) received from the user client terminal 120 in the user table 1200 of the user DB 107.

[0043] 7 shows the intervention measure setting process S700 in this embodiment. The intervention measure setting process S700 in this embodiment is processed by the administrative terminal 130 and the intervention measure management function 108 of the outcome evaluation device 100.

[0044] This process is performed when an administrative official sets an intervention measure. The intervention measure setting process S700 will be described in detail below.

[0045] In process S701, the administrative terminal 130 transmits an administrative login request to the outcome evaluation device 100 via the intervention measure management function 108 provided by the outcome evaluation device 100. At this time, the information transmitted from the administrative terminal 130 includes, for example, an administrative ID assigned to each administration and an administrative password preset for each administration.

[0046] In process S702, the outcome evaluation device 100 performs administrative login authentication based on the login request received from the administrative terminal 130. The login authentication is performed, for example, by comparing the pair of administrative ID and administrative password received from the administrative terminal 130 with the pair of administrative ID and administrative password registered in advance in the outcome evaluation device 100.

[0047] If the administrative login is successful, the process proceeds to step S703, and if it is unsuccessful, the administrative terminal 130 is notified of the login failure and the user is requested to re-enter the administrative ID and administrative password.

[0048] In process S703, the outcome evaluation device 100 displays the intervention measure management screen 800 shown in FIG. 8 on the administrative terminal 130 that has successfully logged in.

[0049] In process S704, the administrative terminal 130 accepts input of a logic model for an intervention measure 810 on an intervention measure management screen 800 provided by the intervention measure management function 108 of the outcome evaluation device 100. The logic model illustrates the relationship between an initial outcome 820, which is a short-term outcome goal that is relatively easy to measure, and a medium- to long-term outcome 830, and is defined by the area surrounded by dotted lines in Figure 8.

[0050] When creating a logic model, items are medium-term outcomes 830 such as the number of people certified as needing assistance or nursing care, and initial outcomes 840 such as the number of people with disabilities in multiple items and the number of people with reduced motor function, and the intervention measure management function 108 accepts input of connecting lines showing the relationships between them and displays the results graphically.

[0051] The initial outcome 820, the mid-term outcome 830, and the relationship between them may reflect knowledge from past academic papers, etc. Furthermore, input of a mathematical formula showing the relationship between the initial outcome 820 and the mid-term outcome 830 based on knowledge from past academic papers, etc. may be accepted.

[0052] If the relationship between the initial outcome 820 and the medium-term outcome 830 can be quantitatively expressed by a mathematical formula, it is possible to find the initial outcome 820 that should be focused on in order to achieve the medium-term outcome 830. Furthermore, input of a measurement method for the initial outcome 820 may be accepted. In process S704, the input for creating a logic model accepted by the administrative terminal 130 is sent to the outcome evaluation device 100.

[0053] In step S705, the intervention management function 108 of the outcome evaluation device 100 receives the logic model 801 input in step S704, and stores the logic model 801 in the outcome table 1400 of the intervention DB 109.

[0054] Process S706 is processed by the administrative terminal 130 as the next process after the logic model creation in process S704. In process S706, the administrative terminal 130 accepts input for setting up an intervention measure on the intervention measure management screen 800 provided by the intervention measure management function 108 of the outcome evaluation device 100.

[0055] The intervention measure 810 is defined in association with an initial outcome 820 of the intervention target, and is displayed graphically on the intervention measure management screen 800 in the same manner as the initial outcome 820 in step S704.

[0056] In addition, by performing a predetermined operation such as double-clicking on an item of the intervention measure 810 (for example, "Intervention measure "P001") on the intervention measure management screen 800, the screen transitions to the intervention measure details screen 900 shown in FIG.

[0057] The intervention measure details screen 900 accepts input of an intervention target outcome 911 of the intervention measure, a subject 912, a measure implementation period 913, an intervention measure 914, an evaluation method 915, a performance-based compensation target 916, a performance-based compensation calculation method 917, evaluation data 918, and an implementing business operator 919. By inputting these, the outcome evaluation device 100 can calculate whether the subject is participating in other intervention measures at the same time, or whether the implementation period overlaps with other intervention measures.

[0058] When a subject is selected for multiple intervention measures at the same time, a relationship like that shown in Figure 3 may be obtained, and propensity score matching or DID analysis can be selected as an evaluation method.

[0059] However, as mentioned above, whether or not subjects participate in an intervention measure is left to their own discretion, and propensity score matching or DID analysis cannot always be performed. Therefore, an analytical method that can be reliably performed, such as a before-and-after comparison, must be set as the next best evaluation method.

[0060] The intervention measure details screen 900 in Figure 9 is designed to allow for manual input of the subjects and implementation times, but it may also be possible to recommend subjects and implementation times so that there is a large overlap between the subjects and implementation times.

[0061] Furthermore, the implementation period of each intervention measure can be clearly indicated in the intervention measure 914 on the intervention measure details screen 900. In the illustrated example, the start and end times of the intervention measures P001 to P003 are different from each other.

[0062] The evaluation method for the intervention measure, the target of the performance compensation, the compensation calculation method, etc. are items required for the outsourcing contract with the business operator, and can be entered, viewed, and modified on the intervention measure details screen 900.

[0063] The evaluation method 915 can select multiple evaluation methods and set priorities according to the level of evidence. In the example shown, propensity score matching, which has the highest level of evidence, is set as priority 1, followed by DID analysis, which has the next highest level of evidence, as priority 2, and before-and-after comparison is set as the evaluation method with the lowest priority.

[0064] Returning to FIG. 7, in step S706, the administrative terminal 130 transmits the input of the intervention policy setting received to the outcome evaluation device 100.

[0065] In step S707, the intervention measure management function 108 of the outcome evaluation device 100 receives the intervention measure input in step S706 and stores it in the intervention measure table 1500 of the intervention measure DB 109.

[0066] By the above process, the intervention measures entered on the intervention measure management screen 800 and the intervention measure details screen 900 are set in the intervention measure table 1500 as shown in Fig. 15, which will be described later. In the example of Fig. 15, the measure ID, outcome, target conditions 1 and 2, target period, evaluation method, and outcome evaluation target (target of performance compensation) of the intervention target are set.

[0067] 10 shows the intervention measure implementation process S1000 in this embodiment. The intervention measure implementation process S1000 in this embodiment is processed by the user client terminal 120 and the user management function 106 of the outcome evaluation device 100. This process is based on the premise that the user login in processes S201 and S202 has been completed.

[0068] In process S1001, the user client terminal 120 obtains a list of intervention measures that are targeted at the logged-in user from the intervention measure table 1500 via the user management function 106 of the outcome evaluation device 100, and displays it on the screen.

[0069] The user client terminal 120 accepts a user's request to apply for an intervention measure and transmits the contents of the request to the outcome evaluation device 100. The request to apply for an intervention measure includes the name of the intervention measure to be applied for, the user's ID, and data for evaluating changes in outcomes due to the intervention measure.

[0070] Data for evaluating changes in outcomes should be easily accessible to users and can be easily entered by users before and after intervention. One example would be the Basic Checklist (Basic CL), which consists of 25 questions and is used in the Ministry of Health, Labour and Welfare's comprehensive program for the elderly. As long as all subjects can provide data before and after intervention, data from health checkups or smart devices would also be acceptable.

[0071] As an example of basic CL, examples of basic CL6 to CL10 are shown in Fig. 18. Fig. 18 shows the checklist number (#), check items, and check result values. In this embodiment, the user client terminal 120 attaches the check results of basic CL6 to CL10 at the time of applying for the policy to the policy application as data before the implementation of the intervention policy as data for evaluating changes in outcomes, and transmits the data to the outcome evaluation device 100.

[0072] In process S1002, the outcome evaluation device 100 receives an intervention measure application request from the user client terminal 120 and stores the contents of the received intervention measure application request in an intervention measure usage table 1300 (described later) of the user DB 107. Note that the intervention measure application is accompanied by the check results of each item of the preset basic CL6 to CL10 as data before the intervention measure is implemented.

[0073] The outcome evaluation device 100 stores the check results of basic CL6 to CL10 assigned to the policy application in the intervention policy utilization table 1300, and sets the status 1304 of the intervention policy utilization table 1300 to "started", as described below.

[0074] The outcome evaluation device 100 may transmit the contents of the intervention measure application request to the administrative terminal 130. Furthermore, if the terminal of the business operator implementing the intervention measure is connected via the network 140, the outcome evaluation device 100 may transmit the contents of the intervention measure application request to the terminal of the business operator.

[0075] Process S1003 is executed when the intervention selected by the user ends. At the end of the intervention, the user client terminal 120 accepts input of data (results of basic CL6 to CL10) for evaluating changes in outcomes due to the intervention, and transmits the data and an application to end the intervention to the outcome evaluation device 100.

[0076] In process S1004, the outcome evaluation device 100 receives the intervention measure termination application from the user client terminal 120 and stores the contents of the received outcome termination application in the intervention measure usage table 1300 of the user DB 107. The outcome evaluation device 100 stores the check results of basic CL6 to CL10 assigned to the policy termination application in the intervention measure usage table 1300, and sets the status 1304 in the intervention measure usage table 1300 to "Ended", as described below.

[0077] The outcome evaluation device 100 may transmit the content of the outcome termination request to the administrative terminal 130. Furthermore, if the terminal of the business operator implementing the intervention measure is connected via a network, the content of the intervention measure termination request may be transmitted to the terminal of the business operator.

[0078] Through the above processing, the outcome evaluation device 100 of the present invention collects data (check results of basic CL6 to CL10) before and after the intervention measure is implemented for the implementer of the intervention measure, and stores the data in the intervention measure utilization table 1300. This processing is executed for each intervention measure implemented by the user.

[0079] When the implementation of the intervention measure is completed, the intervention measure utilization table 1300 stores the check results of the basic CL before the implementation of the intervention measure in basic CLs 6 and 7 (1305, 1306) in the record with status 1304 = "Start", and stores the check results of the basic CL after the implementation of the intervention measure in basic CLs 6 and 7 (1305, 1306) in the record with status 1304 = "End".

[0080] By comparing the values ​​of basic CL6 (1305) and basic CL7 (1306) of records with the same user ID 1302, intervention measure ID 1303, and status 1304 of "Start" and "End" in the intervention measure usage table 1300, the effect of the intervention measure for each basic CL can be obtained.

[0081] 11 shows the intervention measure evaluation process S1100 in this embodiment. The intervention measure evaluation process S1100 in this embodiment is processed by the administrative terminal 130 and the intervention measure management function 108 of the outcome evaluation device 100. This process is based on the premise that the administrative login in processes S701 and S702 in FIG. 7 has been completed.

[0082] In process S1101, the intervention measure management function 108 of the outcome evaluation device 100 displays a list of intervention measures registered in the administrative terminal 130 (or the business operator server 300). The administrative terminal 130 accepts an input from the administrative officer selecting a measure to be evaluated and transmits the content of the selection to the outcome evaluation device 100.

[0083] In step S1102, the intervention measure management function 108 of the outcome evaluation device 100 collects information about the intervention measure to be evaluated, which is specified by the administrative terminal 130, and starts evaluation of the intervention measure using a preset evaluation method.

[0084] First, the intervention measure management function 108 acquires the intervention measure ID 1501, target outcome 1502, target conditions 1 and 2 (1503, 1504), start 1505, end 1506, and performance reward target 1510 as evaluation targets from the intervention measure table 1500 (S1103).

[0085] Next, the intervention measure management function 108 sets the evaluation period from the start 1505 to the end 1506 obtained in the above process S1103 as the evaluation period, and obtains the intervention measure ID 1303 and user ID 1302 implemented during the evaluation period from the intervention measure usage table 1300 (S1104).

[0086] The intervention measure management function 108 first obtains the user ID 1302 and intervention measure ID 1303 of the record from the intervention measure usage table 1300 where the status 1304 is "Start" and the registration date 1307 is within or before the target period, and where the status 1304 is "End" and the registration date 1307 is within or after the target period.

[0087] The intervention measure management function 108 refers to the user ID 1201 in the user table 1200 using the user ID 1302 extracted for the target period, and narrows down the records to be evaluated that satisfy the target condition 1 (1503) related to age. The intervention measure management function 108 acquires the narrowed down records to be evaluated as basic information of the users (S1105).

[0088] If target condition 2 (1504) is set, the intervention measure management function 108 can narrow down the users who satisfy the target condition 2 (1504). In the illustrated example, the condition "medium or higher risk of motor function decline" is set, and this condition corresponds to the "number of people with declined motor function" of outcome ID 1401 = "O003" in the outcome table 1400, and users who meet three or more items among basic CL6 to CL10 are targeted for extraction.

[0089] The intervention measure management function 108 acquires information on basic CL6 (1305) to CL10 for the target period of evaluation from the data of the user ID 1302 extracted for the target period, and sets it as data for intervention measure evaluation (S1106).

[0090] Next, the intervention measure management function 108 combines the data to be evaluated obtained in the above process S1103, the intervention measure and user to be evaluated obtained in the above process S1104, the basic information of the user obtained in the above process S1105, and the data for evaluating the intervention measure obtained in the above process S1106, to generate an analysis dataset 1600 as shown in Figure 16 described below (S1107).

[0091] In the example of Fig. 16, the evaluation target measure 1601 is "intervention measure P003", the evaluation target index 1602 is "basic CL6", and the result (answer) of "basic CL6" is stored as the evaluation target outcome value 1603. Note that the evaluation target outcome value 1603 may be configured to store the result (answer) of "basic CL6" before and after the implementation of the intervention measure.

[0092] The subject of evaluation is intervention measure P003, and the evaluation will assess the impact of this intervention measure on the basic CL6 response (outcome value) of "climbing stairs without holding on to the handrail or wall" before and after the implementation of the intervention measure.

[0093] As shown in intervention plan 914 in Figure 9, at the time intervention plan P003 was implemented, other intervention plans P001 and P002 were also implemented. Therefore, as shown in Figure 3, if there is an area where the users of each intervention plan P001 to P003 overlap (intersect), it becomes possible to use propensity score matching or DID analysis.

[0094] For example, when evaluating the suppressive effect of implementing intervention measure P003 on a person with reduced motor function, information on the user before and after the implementation of the intervention measure is collected from the intervention measure usage table 1300, basic information on the user is collected from the user table 1200, and the evaluation target, evaluation method, and performance-based reward target are collected from the intervention measure table 1500.

[0095] The intervention measure management function 108 performs predetermined statistical processing on the analysis dataset 1600 generated in process S1107 in accordance with the evaluation method (1507 to 1509) of the intervention measure table 1500, and evaluates the intervention measure from the value of the target outcome (evaluation target index) 1502 (number of people with impaired motor function) resulting from the implementation of the intervention measure (S1108).

[0096] The statistical processing performed by the intervention measure management function 108 may be any known or well-known technique such as the propensity score matching, DID analysis, or before-and-after comparison described above, and therefore the details of each statistical processing will not be described in detail.

[0097] Figure 15 shows an example of an intervention measure table 1500 in which the evaluation method and target conditions are set in advance. Evaluation method in Figure 15: In propensity score matching of priority 1 (1507), implementers of intervention measure P003 (1501) are set as the intervention group, and the basic CL (6 to 10) set as the performance-based compensation target 1510 is set as each outcome variable, and the impact of implementing intervention measure P003 on the outcome variable is evaluated by propensity score matching.

[0098] Evaluation method in Figure 15: In the DID analysis of priority 2 (1508), a multiple regression analysis is conducted using the dummy variable Treat, which indicates the implementer of intervention measure P003, the dummy variable Post, which indicates the period after the intervention measure is implemented, and the cross term of Treat x Post, with basic CL (6 to 10) as the dependent variable.

[0099] The dependent variable can be expressed as follows: Dependent variable = α whether or not intervention measures are implemented + β factors Explanatory variable of basic CL = α1 × implementation of intervention measure 1 + α2 × implementation of intervention measure 2 + α3 × implementation of intervention measure 3 + β factor

[0100] It should be noted that intervention measures 1 to 3 correspond to intervention measures P001 to P003, and α1 to α3 and β factors are predetermined coefficients.

[0101] In addition, the explanatory variables for the overlapping areas (intersections) of implementation status with other intervention measures are as follows: Explanatory variable of basic CL = α1 × implementation of measure 1 + α2 × implementation of measure 2 + α3 × (implementation of measure 1 × implementation of measure 2) + α3 × implementation of measure 3 + β factor

[0102] In the example of FIG. 16, the effectiveness of the intervention measure P003 to be evaluated is evaluated from the results of basic CL6, based on whether or not other intervention measures P001 and P002 were implemented.

[0103] The outcome evaluation device 100 evaluates the coefficient of the cross term of Treat×Post as the effect of introducing the intervention measure P003.

[0104] Evaluation method 3: In the comparison before and after Priority 3 (1509), only the implementers of intervention measure P003 are targeted, and the change in basic CL (6-10) before and after implementation is evaluated using a t-test. The evaluation results of these intervention measures P003 are multiplied by the number of implementers to evaluate the compensation for the implementers of the intervention measures.

[0105] The intervention measure management function 108 uses the above evaluation method (1507 to 1509) to evaluate how the risk of motor function decline has changed for users aged 65 to under 75 as a result of implementing intervention measure P003. In other words, the intervention measure management function 108 outputs the number of users whose answers to basic CL6 (evaluation target outcome value 1603) have changed before and after implementing intervention measure P003.

[0106] The intervention measure management function 108 also outputs the number of users whose answers to basic CL6 (evaluation target outcome value 1603) changed, even for users who did not implement intervention measure P003. Users who did not implement intervention measure P003 are users who implemented at least one of intervention measures P001 or P002.

[0107] The intervention measure management function 108 then calculates the number of users with a decline in motor function who implemented intervention measure P003 and the number of users with a decline in motor function who did not implement intervention measure P003, and can therefore indicate whether intervention measure P003 is a significant measure compared to other measures.

[0108] In process S1109, the outcome evaluation device 100 transmits the evaluation result of process S1108 to the administrative terminal 130. The administrative officer using the administrative terminal 130 determines a pre-agreed remuneration for the business operator implementing the intervention measure P003 based on the evaluation result of the intervention measure P003.

[0109] Through the above processing, the outcome evaluation device 100 of this embodiment extracts from the user table 1200 users who have implemented at least one of the multiple intervention measures P001 to P003 whose implementation periods overlap, and performs statistical analysis on the target outcome 1502 for the intervention measure P003 to be evaluated, thereby quantitatively evaluating the effectiveness of the multiple intervention measures P001 to P003.

[0110] When evaluating the effect of intervention measure P003, the effect of intervention measure P003 can be clearly determined by comparing the evaluation target index 1602 between a group of users who have implemented intervention measure P003 and a group of users who have not implemented intervention measure P003. However, it may be difficult for administrative agencies to secure a budget just to measure the evaluation target index 1602 for the group of users who have not implemented intervention measure P003.

[0111] Therefore, the outcome evaluation device 100 of this embodiment can quantitatively evaluate each intervention measure based on the results of the evaluation target index 1602 of multiple intervention measures implemented at the same time, thereby making it possible to determine the effectiveness of the intervention measure to be evaluated.

[0112] In addition, by presenting multiple intervention measures implemented at the same time and obtaining user data, and using the implementation or non-implementation of each intervention measure as a covariate, it becomes possible to calculate the effect of the intervention measure P003 being evaluated. Furthermore, by introducing (or providing) multiple promising intervention measures at the same time and quantitatively evaluating the effect of each intervention measure, truly effective intervention measures can be efficiently introduced.

[0113] The intervention measure management function 108 may output the evaluation results for the evaluation methods (1507 to 1509) in order of priority, or may perform evaluation in order of priority according to the user's participation in the intervention measures.

[0114] Furthermore, the intervention measure management function 108 may present the types of statistical processing such as propensity score matching and DID analysis in descending order of evidence level in the evaluation method 915 of the intervention measure management screen 800 in FIG.

[0115] In addition, the intervention measure management function 108 can present before-and-after comparison as an essential evaluation method in the evaluation method 915 of the intervention measure management screen 800 in Figure 9 as a type of statistical processing when it is difficult to apply propensity score matching or DID analysis.

[0116] Furthermore, as shown in FIG. 3, when multiple users are implementing different intervention measures, the intervention measure management function 108 may preferentially use propensity score matching with a high level of evidence.

[0117] The data used by the outcome evaluation device 100 and the timing of acquiring the data will be described below.

[0118] 12 shows a user table 1200 stored in the user DB 107 in this embodiment. The user table 1200 stores basic information such as the user's name and gender, as well as information related to outcome evaluation, such as family structure and occupation.

[0119] The user table 1200 includes a user ID 1201, a name 1202, a sex 1203, a date of birth 1204, an address 1205, a family structure 1206, and an occupation 1207 in one record.

[0120] In addition to the above, the user table 1200 may also store data on residents held by the government (national health insurance medical receipt data, late-stage elderly health checkup data, certification data on need for support / nursing care, etc.).

[0121] 13 shows an intervention measure usage table 1300 stored in the user DB 107 in this embodiment. The intervention measure usage table 1300 stores data for evaluating an intervention measure that is input by the user at the start and end of the intervention measure. As an example of data for evaluating an intervention measure, this embodiment shows an example in which answer items of basic CL6 to CL10 are stored.

[0122] The intervention measure utilization table 1300 includes an application number 1301, a user ID 1302, an intervention measure ID 1303, a status 1304, a basic CL6 (1305), a basic CL7 (1306), and a registration date 1307 in one record.

[0123] The application number 1301 stores an identifier of the user who applied to participate in the intervention program. The user ID 1302 is an identifier for identifying the user, and stores the value of the user ID 1201 in the user table 1200.

[0124] The intervention measure ID 1303 stores the identifier of the intervention measure selected by the user. The status 1304 stores the participation status of the user in the intervention measure. The basic CL6 (1305) and the basic CL7 (1306) store the answers to the basic CL6 and CL7. Although not shown, the answers to the basic CL4 to CL10 can also be stored. The registration date 1307 stores the year and month (or date and time) when the information was registered.

[0125] With the above configuration, the intervention measure utilization table 1300 stores answer items (answers (1305, 1306) of basic CL6 to CL10) before and after (at the start and end) of each intervention measure for each user, and the answer time (registration year and month 1307). By referencing records with a pair of statuses 1304 of "start" and "end", the intervention measure utilization table 1300 can obtain the user's status before and after the implementation of the intervention measure from the answer items (answers (1305, 1306) of basic CL6 to CL10) as evaluation indicators.

[0126] 14 shows an outcome table 1400 stored in the intervention program DB 109 in this embodiment. The outcome table 1400 includes an outcome ID 1401, an outcome name 1402, a type 1403, and a relationship / evaluation method 1404 in one record.

[0127] The outcome ID 1401 stores an identifier for identifying the outcome. The type 1403 stores the time of the outcome to be acquired. The relationship / evaluation method 1404 stores information related to the relationship between outcomes, the evaluation method, and the definition.

[0128] 15 shows an intervention measure table 1500 stored in the intervention measure DB 109 in this embodiment. The intervention measure table 1500 stores data on the intervention measures set in the user information registration process S600.

[0129] The intervention measure table 1500 includes, in one record, an intervention measure ID 1501, a target outcome 1502, a target condition 1 (1503), a target condition 2 (1504), a start 1505, an end 1506, an evaluation method: priority 1 (1507), an evaluation method: priority 2 (1508), an evaluation method: priority 3 (1509), a performance-based compensation target 1510, a performance-based compensation calculation method 1511, and an implementing company 1512.

[0130] The intervention measure ID 1501 stores an identifier that identifies the intervention measure to be evaluated. The target outcome 1502 stores the name of the outcome to be evaluated (the intervention target outcome 911 in FIG. 9). The target condition 1 (1503) stores the conditions of the person to be evaluated (the subject 912 in FIG. 9). The target condition 2 (1504) stores the name of the outcome to be evaluated (the subject 912 in FIG. 9) and the value. The start 1505 and end 1506 set the period to be evaluated (the policy implementation period 913).

[0131] Evaluation method: priority 1 (1507), evaluation method: priority 2 (1508), and evaluation method: priority 3 (1509) correspond to evaluation method 915 in Figure 9 and store statistical methods for evaluating intervention measures according to priority. Performance-based compensation target 1510 stores the basis for determining the compensation to be paid to the implementing business of the intervention measure (performance-based compensation target 916 in Figure 9). Performance-based compensation calculation method 1511 stores the calculation method for the amount of compensation to be paid to the implementing business (performance-based compensation calculation method 917 in Figure 9). Implementing business 1512 stores the name of the business implementing the intervention measure (implementing business 919 in Figure 9), etc.

[0132] 16 shows an analysis dataset 1600 in this embodiment. The analysis dataset 1600 is data that is temporarily used to search for target data in the intervention measure evaluation process S1100. The analysis dataset 1600 is the user table 1200 with the evaluation target and whether or not an intervention measure has been implemented added.

[0133] The analysis dataset 1600 consists of a group of fields to be evaluated, a group of fields indicating whether or not intervention measures have been implemented, and a group of fields for factors (in this example, user table 1200) that may affect the outcome value 1603 to be evaluated as covariates.

[0134] The group of fields to be evaluated includes an evaluation target measure 1601, an evaluation target index 1602, and an evaluation target outcome value 1603. The evaluation target measure 1601 stores the ID of the intervention measure to be evaluated. This field corresponds to the intervention measure ID 1501 in the intervention measure table 1500. The evaluation target index 1602 stores the index to be evaluated. For example, target condition 1 or 2 in the intervention measure table 1500 is stored. The evaluation target outcome value 1603 stores the value of the target outcome 1502 in the intervention measure table 1500.

[0135] The field group for whether or not an intervention measure has been implemented is shown as an example including implementation status 1604 for P001, implementation status 1605 for P002, and implementation status 1606 for P003. The illustrated example shows an example of intervention measures P001 to P003 displayed in the intervention measure 914 on the intervention measure details screen 900 shown in Fig. 9. Note that in this embodiment, an example has been shown in which the implementation period of the intervention measure P003 to be evaluated overlaps with the implementation periods of the three intervention measures P001 and P002, but the implementation status field for the intervention measures may be set according to the number of intervention measures whose implementation periods overlap.

[0136] The covariate fields include user ID 1607 to occupation 1613, and these fields correspond to the fields in the user table 1200.

[0137] In the illustrated example, an intervention measure P003 is evaluated using the results of basic CL6, and three intervention measures P001 to P003 are used as the analysis targets, and data for each user is combined to create an analysis dataset 1600.

[0138] 17 is a diagram showing the relationship between the timing of data acquisition and whether or not an intervention measure is implemented in this embodiment. In the illustrated example, one user participates in intervention measure P001 at date and time D1, participates in intervention measure P003 at date and time D2, completes intervention measure P003 at date and time D3, and completes intervention measure P001 at date and time D4.

[0139] At date and time D1, the user participates in intervention measure P001 for the first time, so the implementation status of intervention measures P001 to P003 is all marked as "Not implemented." At date and time D2, the user participates in intervention measure P003 for the first time, but has already started intervention measure P001, so intervention measure P001 is marked as "Implemented."

[0140] At date and time D3, intervention measure P003 is terminated, so intervention measures P001 and P003 are marked as "implemented," and the results of basic CL6 to basic CL10 due to intervention measure P003 are stored in the intervention measure utilization table 1300. At date and time D4, intervention measure P001 is terminated, so intervention measures P001 and P003 are marked as "implemented," and the results of basic CL6 to basic CL10 due to intervention measure P001 are stored in the intervention measure utilization table 1300.

[0141] <Conclusion> As described above, the outcome evaluation device 100 of the above embodiment can be configured as follows.

[0142] (1) An outcome evaluation device (100) having a processor (CPU 104) and a memory (102) for evaluating measures (intervention measures) provided by a government, which receives user information (user DB 107) in advance in which the user's status before and after the implementation of each of a plurality of measures is stored as evaluation indicators (basic CL6 (1305), basic CL7 (1306)), measures to be evaluated (intervention measure ID 1501), target outcomes (1502) for evaluating the measures, evaluation target periods (start 1505, end 1506), and evaluation processes (evaluation methods 1507 to 1509), and intervention measure information (intervention measure DB 109) in advance in which the intervention measure information (109) is specified, and evaluates the measures to be evaluated. and an intervention measure management unit (intervention measure management function 108) that evaluates the effectiveness of the measure (1501) to be evaluated using evaluation indexes (1305, 1306) of the user information (107) for the measure (1501), wherein the intervention measure management unit (108) acquires user information (107) of users who implemented at least one of the measures provided during the evaluation target period (1505, 1506) specified in the intervention measure information (109), and evaluates the effectiveness of the measure (1501) to be evaluated using the evaluation indexes (1305, 1306) corresponding to the target outcome (1502) through the evaluation processing (1507-1509).

[0143] With the above configuration, the government can introduce multiple policies for residents at the same time (a designated evaluation period), and the outcome evaluation device 100 can statistically analyze the individual effects of the multiple policies. The ability to quantitatively evaluate the effects of policies supports the introduction of PFS-type businesses and contributes to the promotion of efficient policies that utilize the private sector.

[0144] (2) An outcome evaluation device according to (1) above, characterized in that the intervention policy management unit (108) evaluates the effectiveness of the policy (1501) being evaluated by using the implementation or non-implementation of the multiple policies (implementation or non-implementation of P001 1604, implementation or non-implementation of P002 1605, implementation or non-implementation of P003 1606) as covariates.

[0145] With the above configuration, it is possible to calculate the effect of the intervention measure P003 to be evaluated by performing statistical processing using the implementation or non-implementation of multiple measures provided during the specified evaluation period as a covariate.

[0146] (3) The performance evaluation device described in (1) above, wherein the evaluation process (1507-1509) for evaluating the policy (1501) to be evaluated is a performance evaluation device characterized in that multiple types of evaluation processes (1507-1509) with priorities assigned are pre-set, and the policy is evaluated using the multiple types of evaluation processes (1507-1509) in order of priority.

[0147] With the above configuration, by assigning priorities to the evaluation processes, it becomes possible to apply evaluation processes of analytical accuracy according to the measures to be evaluated.

[0148] (4) The performance evaluation device described in (1) above, wherein the user information (107) includes the attributes of the user (user ID 1201, name 1202, gender 1203, date of birth 1204, address 1205, family composition 1206, occupation 1207), and specifies the attributes (1201 to 1207) of the user who implemented at least one of the measures (P001 to P003) provided during the evaluation period (1505, 1506).

[0149] With the above configuration, by performing desired filtering on the user information used in the statistical processing of the policy to be evaluated, it becomes possible to perform evaluation using data that corresponds to the purpose of the policy to be evaluated.

[0150] (5) The outcome evaluation device according to (3) above, wherein the plurality of types of evaluation processes (1507 to 1509) are set in descending order of evidence level.

[0151] With the above configuration, by assigning priorities to the evaluation processes in descending order of evidence level, it becomes possible to carry out the evaluation process at the evidence level desired by the user of the outcome evaluation device.

[0152] (6) The outcome evaluation device according to (3) above, wherein the plurality of types of evaluation processes (1507 to 1509) include propensity score matching or differential analysis of differences.

[0153] With the above configuration, in propensity score matching, those who implemented intervention measure P003 (1501) are set as the intervention group, and the basic CL (6 to 10) set as the performance-based compensation target 1510 is set as each outcome variable, and propensity score matching can be used to evaluate the impact of implementing intervention measure P003 on the outcome variable. Furthermore, in difference-in-differences analysis, multiple regression analysis can be performed using the dummy variable Treat, which indicates those who implemented intervention measure P003, the dummy variable Post, which indicates after the implementation of the intervention measure, and the cross term of Treat x Post, with the basic CL (6 to 10) each as an explained variable.

[0154] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, with respect to part of the configuration of each embodiment, addition, deletion, or substitution of other configurations can be applied alone or in combination.

[0155] Furthermore, the above-described configurations, functions, processing units, and processing means may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations and functions may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0156] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0157] 100 Performance Evaluation System 101, 121, 131 I / O interfaces 102, 122, 132 memory 103, 123, 133 Communication equipment 104, 124, 134 CPU (computing unit) 105, 125, 135 storage device 106 User management function 107 User Database 108 Intervention measure management function 109 Intervention Policy Database

Claims

1. A performance evaluation device having a processor and a memory for evaluating policies provided by a government, User information in which the user's status before and after the implementation of each of a plurality of measures is stored in advance as an evaluation index; Intervention policy information that specifies a policy to be evaluated, a target outcome for evaluating the policy, a period to be evaluated, and an evaluation process in advance; an intervention measure management unit that receives the intervention measure information and evaluates the effectiveness of the measure to be evaluated using the evaluation index of the user information for the measure to be evaluated, The intervention policy management unit An outcome evaluation device characterized by acquiring user information of users who implemented at least one of the measures provided during the evaluation period specified in the intervention measure information, and evaluating the effectiveness of the measure to be evaluated using the evaluation index corresponding to the target outcome.

2. The outcome evaluation device according to claim 1, The intervention policy management unit An outcome evaluation device characterized by evaluating the effectiveness of the measures to be evaluated by using whether or not the multiple measures are implemented as a covariate.

3. The outcome evaluation device according to claim 1, The evaluation process for evaluating the policy to be evaluated includes: A performance evaluation device characterized in that multiple types of evaluation processes with priorities assigned are pre-set, and the measures are evaluated using the multiple types of evaluation processes in order of priority.

4. The outcome evaluation device according to claim 1, The user information is An outcome evaluation device including attributes of the user, characterized in that the device specifies attributes of a user who implemented at least one of the measures provided during the evaluation period.

5. The outcome evaluation device according to claim 3, An outcome evaluation device characterized in that the multiple types of evaluation processes are set in order of increasing evidence level.

6. The outcome evaluation device according to claim 3, The outcome evaluation device, wherein the multiple types of evaluation processing include propensity score matching or difference-in-difference analysis.

7. A performance evaluation method in which a computer having a processor and a memory evaluates a policy provided by a government, comprising: a first step in which the computer acquires user information stored in advance, the user's status before and after implementation of each of a plurality of measures being used as an evaluation index; a second step in which the computer acquires intervention policy information that pre-specifies a policy to be evaluated, a target outcome for evaluating the policy, a period to be evaluated, and an evaluation process; a third step in which the computer receives the intervention measure information and evaluates the effectiveness of the measure to be evaluated using the evaluation index of the user information for the measure to be evaluated, In the third step, An outcome evaluation method characterized by obtaining user information of users who implemented at least one of the measures provided during the evaluation period specified in the intervention measure information, and evaluating the effectiveness of the measure to be evaluated using the evaluation process based on the evaluation index corresponding to the target outcome.

8. The outcome evaluation method according to claim 7, In the third step, An outcome evaluation method characterized by evaluating the effectiveness of the measures being evaluated by using whether or not the multiple measures are implemented as a covariate.

9. The outcome evaluation method according to claim 7, The evaluation process for evaluating the policy to be evaluated includes: A performance evaluation method characterized in that multiple types of evaluation processes with priorities assigned are pre-set, and the measures are evaluated using the multiple types of evaluation processes in order of priority.

10. The outcome evaluation method according to claim 7, The user information is An outcome evaluation method including attributes of the user, characterized in that the attributes of the user who implemented at least one of the measures provided during the evaluation period are specified.

11. The outcome evaluation method according to claim 9, An outcome evaluation method characterized in that the multiple types of evaluation processes are set in order of increasing evidence level.

12. The outcome evaluation method according to claim 9, The outcome evaluation method, wherein the multiple types of evaluation processing include propensity score matching or difference-in-difference analysis.

Images