Performance evaluation method for digital informatization project

Abstract

The invention discloses a digital informatization project performance evaluation method. The method comprises the following steps: constructing an index system according to historical data of a digital informatization project; performing weight assignment on each index in the constructed index system based on an index weight assignment method to obtain an index assignment result; based on an evaluation data acquisition method, rechecking and confirming the obtained index assignment result; and performing comprehensive analysis on the evaluation content characteristics of the digital informatization project based on an evaluation scoring method, and outputting a comprehensive analysis result. According to the method, the index system is constructed, and project evaluation is performed based on the corresponding evaluation method, so that multi-level, multi-dimensional and multi-view analysis is performed on the digital informatization projects in different types and different fields, the data acquisition method is efficient and reliable, and the evaluation index pertinence is relatively high.

Description

Technical Field

[0001] The present invention relates to the technical field of project performance evaluation, and in particular to a digital information project performance evaluation method. Background Art

[0003] With the continuous advancement of digital construction, while various investments are continuously flowing into the construction of digital project infrastructure and application systems, the value produced is also in urgent need of scientific evaluation. In recent years, various indicator systems or evaluation frameworks for digital projects have emerged. Some frameworks are aimed at evaluating the public value of digital projects, and some frameworks are evaluated based on the characteristics of websites. At present, it is particularly important to establish a universal digital information project performance evaluation system. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, a digital information project performance evaluation method is adopted to solve the problems raised in the above background technology.

[0005] A digital information project performance evaluation method includes the following steps:

[0006] Step S1, constructing an indicator system based on historical data of digital information projects;

[0007] Step S2: assigning weights to each indicator in the constructed indicator system based on the indicator weight assignment method to obtain indicator assignment results;

[0008] Step S3: Based on the evaluation data collection method, the obtained indicator assignment results are reviewed and confirmed;

[0009] Step S4: Perform a comprehensive analysis on the evaluation content characteristics of the digital informationization project based on the evaluation scoring method, and output the comprehensive analysis results.

[0010] As a further solution of the present invention: the specific steps in step S1 include:

[0011] Step S11, define the indicator system:

[0012] Multiple indicators are integrated according to the preset relationship, and the three elements of the indicator system are set, including the parent indicator, the sub-indicators, and the relationship between the indicators;

[0013] Step S12: construct an indicator system:

[0014] Based on horizontal expansion decomposition, vertical subdivision decomposition, and the establishment of a hierarchical indicator system, the parent indicator is decomposed into multiple sub-indicators to form an indicator system.

[0015] As a further solution of the present invention: the specific steps in step S2 include:

[0016] Step S21: assigning a weight to each indicator in the constructed indicator system based on an indicator weight assignment method, wherein the indicator weight assignment method includes a subjective assignment method and an objective assignment method;

[0017] Step S22: Determine the weights in a qualitative manner based on subjective value assignment methods, including Delphi method, AHP method and rating scoring method;

[0018] Step S23: Based on objective value assignment methods, including entropy method, principal component analysis method, and grey correlation method, weights are determined according to the characteristics or relationships of the data itself.

[0019] As a further solution of the present invention: the specific steps in step S3 include:

[0020] To obtain evaluation perspectives from multiple dimensions, based on the feasibility and sustainability of the evaluation, the evaluation data is collected mainly using four methods: document verification, system verification, user interviews, and questionnaire surveys, and the indicator assignment results are reviewed and confirmed.

[0021] As a further solution of the present invention: the specific steps in step S4 include:

[0022] Step S41, comprehensively analyzing the evaluation content characteristics of the digital informationization project based on the evaluation scoring method, wherein the evaluation scoring method includes the evaluator scoring method, the success evaluation method, the field investigation method, the before-and-after comparison method, the presence-and-absence comparison method, the logical framework method, and the hierarchical analysis method;

[0023] Step S42: According to the characteristics and evaluation requirements of the digital informationization project, one or more evaluation and scoring methods are selected to conduct performance evaluation on the project, and the evaluation results are output.

[0024] Compared with the prior art, the present invention has the following technical effects:

[0025] The above technical solution, through the construction of the index system of digital information project performance evaluation, provides guidance for the performance evaluation work, helps to ensure the scientificity, standardization and practical effectiveness of the evaluation work, helps to accurately measure the status, efficiency and effect of digital information projects, and timely discovers implementation deviations. It will help the competent departments to further optimize their work measures, promote the level of information construction to a higher level, promote the rapid, healthy and orderly development of local digital information projects, and enhance the competitiveness of digital information. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0027] Figure 1 A schematic diagram of the steps of the performance evaluation method disclosed in the embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Please refer to Figure 1 In an embodiment of the present invention, a digital information project performance evaluation method includes the following steps:

[0030] Step S1: construct an indicator system based on the historical data of the digital information project. The specific steps include:

[0031] Step S11, define the indicator system:

[0032] Multiple indicators are integrated according to the preset relationship, and the three elements of the indicator system are set, including the parent indicator, the sub-indicators, and the relationship between the indicators;

[0033] Step S12: construct an indicator system:

[0034] Based on horizontal expansion decomposition, vertical subdivision decomposition, and the establishment of a hierarchical indicator system, the parent indicator is decomposed into multiple sub-indicators to form an indicator system.

[0035] In this embodiment, the commonly used disassembly method is:

[0036] (I) Decomposition according to business process: According to the various stages of the business process, the indicator system is divided into corresponding sub-indicators.

[0037] (II) User life cycle analysis: With users at the center, the indicator system is divided into corresponding sub-indicators based on the behaviors and needs of users at different stages of the life cycle. For example, key indicators are set separately in the user acquisition, activation, retention, revenue and recommendation stages.

[0038] (III) Disassembly using OSM model: OSM model includes objectives, strategies and measures. First, determine the business objectives, then formulate strategies to achieve the objectives, and finally break down these strategies into measurable indicators. In this way, the indicator system can be disassembled layer by layer according to objectives, strategies and measures.

[0039] (IV) Decomposition by product functional modules: According to the functional modules of products or services, the indicator system is divided into sub-indicators of each module.

[0040] (V) Disassembly based on the pyramid principle: The pyramid principle is a way of thinking from top to bottom, which can help us clarify the logical relationship between indicators. When building an indicator system, we can start with the top-level goals and gradually disassemble them into middle-level strategies and bottom-level specific indicators.

[0041] (6) Learn from the industry's mature decomposition models: such as UJM (User Journey Map), AARRR (Acquisition, Activation, Retention, Revenue, Referral) and other models. These models provide a structured approach that can help us better decompose the indicator system.

[0042] Step S2: assigning weights to each indicator in the constructed indicator system based on the indicator weight assignment method to obtain indicator assignment results. The specific steps include:

[0043] Step S21: assigning a weight to each indicator in the constructed indicator system based on an indicator weight assignment method, wherein the indicator weight assignment method includes a subjective assignment method and an objective assignment method;

[0044] Step S22: Determine the weights in a qualitative manner based on subjective value assignment methods, including Delphi method, AHP method and rating scoring method;

[0045] In this embodiment, three common methods are compared as shown in Table 1 below:

[0046] Table 1 Comparison of subjective assignment methods

[0047]

[0048] Step S23: Based on objective value assignment methods, including entropy method, principal component analysis method, and grey correlation method, weights are determined according to the characteristics or relationships of the data itself.

[0049] In this embodiment, common methods include entropy method, principal component analysis method, grey correlation method, etc. For example, entropy method uses information entropy redundancy to determine the weight of indicators. The greater the information entropy redundancy, the greater the indicator weight. The weight of each indicator can be obtained by calculating the ratio of the information entropy redundancy of the indicator to the sum of the information entropy redundancy of all indicators, where the sum of the weights of all indicators is 1. The detailed comparison of several methods is shown in Table 2 below:

[0050] Table 2 Comparison of objective assignment methods

[0051]

[0052] Step S3: Based on the evaluation data collection method, the obtained indicator assignment results are reviewed and confirmed. The specific steps include:

[0053] To obtain evaluation perspectives from multiple dimensions, based on the feasibility and sustainability of the evaluation, the evaluation data is collected mainly using four methods: document verification, system verification, user interviews, and questionnaire surveys, and the indicator assignment results are reviewed and confirmed.

[0054] In this embodiment, the data checking includes the following specific steps:

[0055] Review relevant documents on project approval, procurement, management, and acceptance, conduct on-site inspections of the authenticity of materials, completeness of content, comparative analysis of demand design and implementation effects, and evaluate the standardization of the project construction process;

[0056] In this embodiment, the system verifies, and the specific steps are:

[0057] By logging into the system backend account, or by the staff's demonstration, check the system service performance, business support, user experience, etc. In terms of application services, check the system user registration status, access activity, etc.; in terms of hardware use, check the system CPU utilization, physical memory utilization, storage utilization, etc., and check the use of special equipment for special equipment; operation and maintenance management check fault handling, inspection records, system logs, etc.; security management check system implementation, data security mechanism, password application management, computer room security (if any), etc.;

[0058] In this embodiment, the user interview includes the following specific steps:

[0059] In this embodiment, a questionnaire survey is conducted around the actual use of the project, service performance, supported business conditions, system shared application conditions, etc. The specific steps are as follows:

[0060] The specific project questionnaire is designed with non-scale items as the main items. It is set in combination with the actual situation of the project, and the status quo and sample attitude of a certain project are understood. Relevant suggestions are provided through the questionnaire. The survey subjects are divided into two categories: system staff and external service objects. The sampling quantity is ≥50 for system staff and ≥100 for external service objects.

[0061] Step S4: Comprehensively analyze the evaluation content characteristics of the digital informationization project based on the evaluation scoring method and output the comprehensive analysis results. The specific steps include:

[0062] Step S41, comprehensively analyzing the evaluation content characteristics of the digital informationization project based on the evaluation scoring method, wherein the evaluation scoring method includes the evaluator scoring method, the success evaluation method, the field investigation method, the before-and-after comparison method, the presence-and-absence comparison method, the logical framework method, and the hierarchical analysis method;

[0063] In this embodiment, through comprehensive analysis of the evaluation content characteristics, the evaluation scoring methods are summarized into the following seven types:

[0064] 1. The evaluator scoring method: the evaluator directly collects data from the platform or system of the evaluation object, or queries relevant data through the system operation log, obtains data, assesses scores, and evaluates effectiveness;

[0065] Second, the success evaluation method is mainly based on expert experience, through watching system demonstrations, listening to situation reports, reading materials, etc. to score, assess scores, and evaluate effectiveness. The expert scoring method can be combined with the evaluator scoring method to conduct performance evaluation on the same system. It mainly gives a qualitative conclusion on the degree of success or failure of the project in achieving the expected goals. It can be distinguished by five levels: complete success, success, partial success, unsuccessful, and failure;

[0066] 3. Field investigation method: According to the purpose of the evaluation, the system is investigated and studied through on-site interviews, research checklists, online seminars, expert consultations, etc., to obtain data and information and evaluate the effectiveness. For systems or projects that are difficult to score objectively, the investigation method can be used.

[0067] Fourth, the before-after comparison method is a method of comparing the situation after the project implementation with the expected goals before the project implementation to determine the project benefits. It mainly examines the difference between the actual implementation of the project and the expected situation, and evaluates the decision-making level and management ability of investors. The before-after comparison method analysis is mainly used for project goal evaluation, especially for business goal evaluation, network goal evaluation, income goal evaluation and progress evaluation. Such evaluations can generally be compared before and after with quantitative indicators.

[0068] 5. The method of comparison between the actual situation of the project and the situation that might occur if there is no project, in order to measure the real benefits, effects and impacts of the project. The key point of the comparison is to distinguish the effects and impacts of the project from the effects and impacts of factors outside the project. This comparison is used for the benefit evaluation and impact evaluation of the project, and is also an important method of post-evaluation. The key to the method of comparison between the cost of the input and the effect of the output is to require the consistency of the caliber of the effect, so that the measured effect can be truly attributed to the project. The premise and assumption of its application is that the effect of factors outside the project is zero. In fact, this situation is generally an ideal state, which is generally defined as the effect of factors outside the project is small or negligible.

[0069] 6. Logical framework method is a comprehensive evaluation method that analyzes and summarizes projects through four levels: input, output, direct purpose, and macro impact. It is a method of conceptualizing projects, that is, using a simple frame diagram to clearly analyze the connotation and relationship of a complex project to make it easier to understand. The logical framework method combines several dynamic factors that are related and must be considered simultaneously, and evaluates an activity or work by analyzing the relationship between them, from design planning to the determination of goals and objectives. The logical framework method provides an analysis framework for project planners to determine the scope and tasks of the work by analyzing the logical relationship between project goals and the means required to achieve the goals.

[0070] 7. The analytic hierarchy process (AHP) treats a complex multi-objective decision-making problem as a system, decomposes the goal into multiple goals or criteria, and then decomposes it into several levels of multiple indicators (or criteria, constraints). Then, the method of solving the eigenvector of the judgment matrix is ​​adopted to obtain the priority weight of each element at each level to an element at the previous level. Finally, the final weight of each goal to the total goal is recursively merged by reweighted summation, as a systematic method for optimizing decision-making with multiple goals (multi-indicators) and multiple programs.

[0071] Step S42: According to the characteristics and evaluation requirements of the digital informationization project, one or more evaluation and scoring methods are selected to conduct performance evaluation on the project, and the evaluation results are output.

[0072] In this embodiment, the specific evaluation method can select one or more methods to evaluate the performance of the project according to the characteristics and evaluation requirements of the digital information project. Since different methods have different focuses and the difficulty of evaluation varies greatly, the corresponding method should be selected for different projects.

[0073] For projects where the indicator data required for evaluation are not easy to obtain, especially for indicators that are difficult to quantify, the expert scoring method can be directly used for evaluation; for some investment projects with obvious benefits, large differences before and after the project or with or without the project, and where the data before and after the project can be collected, the comparison method should be used; for those projects that are relatively complex, have large investment amounts, and are of great social significance, the logical framework method should be used to conduct a comprehensive evaluation of the projects.

[0074] In the target evaluation, the before-after comparison method is basically used for the evaluation of project targets, that is, the actual achieved value is compared with the project's pre-estimated value. In terms of benefit target evaluation, the benefit evaluation method is to compare the actual benefits or effects of the project with the actual costs or inputs incurred by the project for analysis. In the project performance evaluation stage, the calculation of benefit indicators is completely based on the actual statistical values ​​for statistical analysis, and the same economic evaluation parameters used in the prior evaluation are used for benefit calculation accordingly, so as to judge whether the decision of the project is correct under the condition of comparability and consistent calculation caliber. By comparing the two, the degree of goal achievement is evaluated, and the reasons for the difference are further analyzed.

[0075] During the process evaluation, the expert scoring method can be used to evaluate the success of the entire project process. Through the analysis and evaluation of the entire process, the difference between subjective wishes and reality can be found, and the main links and reasons leading to the success or failure of the project can be discovered, and relevant suggestions and measures can be put forward to make the implementation plans and goals of similar projects in the future more practical and feasible.

[0076] Since the expert scoring method is relatively universally applicable and has a certain degree of systematicity, it is operational for the overall project evaluation. On the basis of the application success evaluation method, the before-and-after comparison method, the logical framework method, and the presence / absence comparison method can be applied in combination with the specific circumstances and characteristics of the project.

[0077] In general, the performance evaluation of digital information projects adopts a combination of quantitative and qualitative analysis, mainly using a comprehensive indicator system evaluation method. For evaluation indicators that can collect data, quantitative analysis methods should be used as much as possible in principle. For evaluation indicators that cannot collect data and cannot directly measure benefits, they can be evaluated through methods such as data review, expert evaluation, public questionnaires, and sampling surveys, and converted into quantifiable indicators for evaluation.

[0078] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.

Claims

1. A digital information project performance evaluation method, characterized in that: The following steps are involved: Step S1, constructing an indicator system based on historical data of digital information projects; Step S2: assigning weights to each indicator in the constructed indicator system based on the indicator weight assignment method to obtain indicator assignment results; Step S3: Based on the evaluation data collection method, the obtained indicator assignment results are reviewed and confirmed; Step S4: Perform a comprehensive analysis on the evaluation content characteristics of the digital informationization project based on the evaluation scoring method, and output the comprehensive analysis results.

2. A digital information project performance evaluation method according to claim 1, characterized in that: The specific steps in step S1 include: Step S11, define the indicator system: Multiple indicators are integrated according to the preset relationship, and the three elements of the indicator system are set, including the parent indicator, the sub-indicators, and the relationship between the indicators; Step S12: construct an indicator system: Based on horizontal expansion decomposition, vertical subdivision decomposition, and the establishment of a hierarchical indicator system, the parent indicator is decomposed into multiple sub-indicators to form an indicator system.

3. A digital information project performance evaluation method according to claim 1, characterized in that: The specific steps in step S2 include: Step S21: assigning a weight to each indicator in the constructed indicator system based on an indicator weight assignment method, wherein the indicator weight assignment method includes a subjective assignment method and an objective assignment method; Step S22: Determine the weights in a qualitative manner based on subjective value assignment methods, including Delphi method, AHP method and rating scoring method; Step S23: Based on objective value assignment methods, including entropy method, principal component analysis method, and grey correlation method, weights are determined according to the characteristics or relationships of the data itself.

4. A digital information project performance evaluation method according to claim 1, characterized in that: The specific steps in step S3 include: To obtain evaluation perspectives from multiple dimensions, based on the feasibility and sustainability of the evaluation, the evaluation data is collected mainly using four methods: document verification, system verification, user interviews, and questionnaire surveys, and the indicator assignment results are reviewed and confirmed.

5. A digital information project performance evaluation method according to claim 1, characterized in that: The specific steps in step S4 include: Step S41, comprehensively analyzing the evaluation content characteristics of the digital informationization project based on the evaluation scoring method, wherein the evaluation scoring method includes the evaluator scoring method, the success evaluation method, the field investigation method, the before-and-after comparison method, the presence-and-absence comparison method, the logical framework method, and the hierarchical analysis method; Step S42: According to the characteristics and evaluation requirements of the digital informationization project, one or more evaluation and scoring methods are selected to conduct performance evaluation on the project, and the evaluation results are output.

Images