Dynamic multi-level evaluation system

Abstract

The invention provides a dynamic multi-level evaluation system, which comprises an evaluation specification management module used for storing an evaluation plug-in integration specification, an evaluation data description specification, an evaluation plug-in development specification and an evaluation model description specification; the evaluation data management module is used for creating an evaluation data model required by the task based on the evaluation data description specification; the evaluation model management module is used for completing the design of a comprehensive calculation model on the basis of an evaluation model which is developed by the evaluation model description specification; the evaluation plug-in management module is used for managing an evaluation plug-in, and the evaluation plug-in is packaged according to an evaluation plug-in integration specification and is called by the evaluation model management module; the evaluation system management module is used for designing and managing a multi-level index system required by an evaluation task; and the evaluation scheme management module selects a multi-level index system based on the evaluation task to create an evaluation scheme, and executes the evaluation scheme to generate an evaluation result. The method is clear in logic and simple to operate, and the evaluation work efficiency is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of evaluation, and in particular to a dynamic multi-level evaluation system. Background Technology

[0002] The process of constructing an indicator system refers to breaking down complex evaluation objects into specific, quantifiable evaluation indicators. Constructing a scientific and reasonable indicator system can provide unified standards and norms for evaluation activities, effectively reducing evaluation biases caused by subjective factors or arbitrariness. Simultaneously, it can provide theoretical and technical support for specific fields, promoting in-depth research and practice in those fields. A scientific indicator system can significantly improve the efficiency, accuracy, and guiding role of evaluation work, providing strong support for decision-making and management, thereby providing decision-makers with comprehensive and objective data.

[0003] Currently available literature presents several methods for constructing indicator systems. For example, the paper "Ideas and Suggestions on the Selection of Policy Evaluation Methods and the Construction of Evaluation Indicator Systems in my country" proposes a hybrid evaluation framework combining quantitative and qualitative methods, emphasizing a combination of subjective and objective weighting. A joint research group from multiple universities, in their study "Construction of Performance Evaluation Indicator System for Excellent Young Scientists Fund Projects," uses the Delphi method to screen indicators and analytic hierarchy process (AHP) to determine weights, constructing a three-tiered indicator system encompassing "quality of scientific research achievements," "academic influence," and "team building." Guangxi Power Grid Co., Ltd. has applied for a patent titled "A Method for Constructing a Carbon Assessment Classification Indicator System for Zero-Carbon Power Supply Stations Considering Carbon Flow Analysis," publication number CN118941135A. This invention discloses a method for constructing a carbon assessment classification indicator system for zero-carbon power supply stations considering carbon flow analysis. Based on the output results and the goals of zero-carbon power supply stations, various carbon assessment classification indicators are obtained, and weights are assigned to these indicators. The weighted carbon assessment analysis indicators are then integrated to obtain the carbon assessment classification indicator system for zero-carbon power supply stations. China Steel Wuhan Safety and Environmental Protection Research Institute Co., Ltd. has applied for a patent entitled "A Method for Constructing and Evaluating a Zero-Carbon Index System for Industrial Parks", publication number CN119273204A. This patent discloses a method for constructing and evaluating a zero-carbon index system for industrial parks, which constructs a zero-carbon index system for industrial parks from four dimensions; determines the indicators at each level of different dimensions, uses the analytic hierarchy process and entropy weight method to determine the weight of the indicators, and finally calculates the comprehensive score.

[0004] Existing literature on indicator systems primarily focuses on theoretical construction. While numerous methodologies and frameworks have been proposed, a flexible and practical system is lacking to validate the construction and application of indicator systems in real-world business. This lack of a system hinders the effective integration of data, models, and indicator systems. Significant gaps remain regarding how to combine data with models, how to achieve data-driven indicator system construction, and how to validate model effectiveness through data. A complete closed loop cannot be formed, making it difficult to obtain reliable results through practical operation and verification. This situation not only limits the practical application value of indicator systems but also affects the efficiency and accuracy of evaluation work. There is an urgent need to develop a system that can support the integration of data and models and flexibly adapt to different needs to solve this problem. Summary of the Invention

[0005] This application provides a dynamic multi-level evaluation system to address the problems in the background art.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] According to a first aspect of the embodiments of this application, a dynamic multi-level evaluation system is provided, comprising: The assessment specification management module is used to store assessment plugin integration specifications, assessment data description specifications, assessment plugin development specifications, and assessment model description specifications. The assessment data management module creates the assessment data model required for the task based on the assessment data description specification. The evaluation model management module develops evaluation models based on the evaluation model description specification and calls evaluation plugins to complete the design of the evaluation models. The evaluation plugin management module is used to manage evaluation plugins developed based on the evaluation plugin development specification. The evaluation plugins are encapsulated according to the evaluation plugin integration specification and are called by the evaluation model management module. The assessment system management module is used to design and manage the multi-level indicator system required for assessment tasks and to link it with the assessment model; The evaluation scheme management module allows you to create an evaluation scheme by selecting a multi-level indicator system based on the evaluation task. Executing the evaluation scheme will generate the evaluation results.

[0008] According to one embodiment of this application, the evaluation plug-in integration specification, evaluation data description specification, evaluation plug-in development specification, and evaluation model description specification are all formed based on the collation and analysis of evaluation task documents, evaluation system, and evaluation indicators.

[0009] According to one embodiment of this application, the evaluation data management module further includes a preprocessing model for preprocessing the evaluation data.

[0010] According to one embodiment of this application, the evaluation model management module includes an evaluation model developed based on an evaluation data model and a comprehensive calculation model designed by combining the evaluation model and the evaluation query plugin.

[0011] According to one embodiment of this application, the design of the multi-level indicator system includes: determining the hierarchical structure of the indicator system and determining the specific indicators included in each level. The indicators are divided into final-level indicators and comprehensive calculation indicators. The final-level indicators are associated with the evaluation data model and the evaluation model. The comprehensive calculation indicators are bound to the designed comprehensive calculation model.

[0012] According to one embodiment of this application, the evaluation scheme execution process in the evaluation scheme management module includes: Obtain evaluation plan information and generate an execution diagram; Convert the nodes in the execution graph into execution sequences, and execute the nodes in the sequences sequentially; Save the calculation results of each node and output the evaluation results.

[0013] According to one embodiment of this application, the specific execution process of any node in the execution sequence includes: Obtain the evaluation model information from the nodes and determine whether it is a comprehensive calculation model; If it is not a comprehensive calculation model, the input data is obtained, and the data preprocessing model is called to preprocess the data according to business needs, or the evaluation model is executed directly, and the upstream result data generated by the execution is saved to the database; when executing the evaluation model, if the evaluation model comes from an external plugin, the corresponding evaluation plugin is executed directly, otherwise the evaluation model is executed directly. If it is a comprehensive calculation model, then the upstream result data is obtained, and the comprehensive calculation index result is calculated.

[0014] According to one embodiment of this application, after saving the calculation results of each node, the method further includes: Determine whether the evaluation plan is configured for periodic execution; If the configuration period is executed, then according to the configured period, the nodes in the sequence will be re-evaluated sequentially. If periodic execution is not configured, it further determines whether real-time execution is required. If real-time execution is required, the current evaluation scheme is executed repeatedly; otherwise, the current evaluation scheme is executed only once.

[0015] According to one embodiment of this application, obtaining input data specifically includes: The data model is evaluated to determine whether the source of the input data is a file, a database, or an API interface. If the data is determined to be from a file, it is retrieved from the file storage; if the data is determined to be from a database, it is retrieved from the database; otherwise, the data is retrieved through the API interface.

[0016] According to one embodiment of this application, after generating the execution graph, it is determined whether there is an infinite loop in the execution graph. If it exists, an error message is displayed; if it does not exist, subsequent operations are performed.

[0017] Compared with existing technologies, the beneficial effects of adopting the above technical solution are as follows: Based on existing methodologies and frameworks for indicator systems, this invention develops a flexible and practical system for constructing and applying evaluation systems, enabling the effective integration of data, models, and indicator systems. The system is logically clear, easy to operate, and significantly improves the efficiency of evaluation work. This invention mainly has the following advantages: (1) Scenario adaptability. Through the design of a multi-level indicator system and the integration mechanism of quantitative and qualitative indicators, the system can flexibly adapt to the assessment needs of different fields such as finance, energy, and healthcare.

[0018] (2) Dynamic Adjustment Capability. The system supports the free splitting and recombination of evaluation indicators, allowing for flexible selection of required indicators based on specific business needs, and real-time adjustment and optimization of existing indicators. Furthermore, this method allows for dynamic correction of indicator parameters, ensuring the accuracy and adaptability of the evaluation. The built-in expert scoring mechanism can be used flexibly according to actual needs; the system also supports hybrid evaluation models, enabling both data-driven automatic evaluation and comprehensive evaluation combining expert scoring. Simultaneously, it provides a dynamic adjustment function for expert weights, allowing for flexible adjustment of the level of expert participation based on different evaluation scenarios, further enhancing the flexibility and accuracy of the evaluation. (3) Flexibility and Scalability. The multi-dimensional and multi-level indicator system constructed by this system can flexibly adjust the weights, data sources, and evaluation algorithms of indicators according to different business needs and application scenarios. At the same time, the flexible design of the system allows for the addition, deletion, and modification of indicators, ensuring that the indicator system can be dynamically updated with business development and technological progress. In addition, through defined specifications, open interfaces, and standardized data formats, the system supports seamless integration with other systems and tools, and can be easily extended to more complex analysis scenarios, such as the introduction of real-time computing, big data analysis, and machine learning models. This flexibility and scalability not only ensures the applicability of the indicator system in the current scenario, but also provides a solid foundation for future optimization and upgrades. Attached Figure Description

[0019] Figure 1This is a schematic diagram of the dynamic multi-level evaluation system according to an embodiment of this application.

[0020] Figure 2 This is a flowchart illustrating the evaluation scheme execution of an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar modules or modules having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. Rather, the embodiments of this application include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0022] To address the shortcomings of existing solutions, this application proposes a dynamic multi-level evaluation system. This system can adapt to different scenario requirements, supports dynamic adjustment and optimization, and can flexibly select appropriate construction methods and models according to the characteristics of different data and the requirements of evaluation objectives. The system can achieve deep integration of data and models and support the construction of data-driven indicator systems.

[0023] Furthermore, the dynamic multi-level evaluation system proposed in this embodiment possesses a high degree of customization and dynamic adjustment capabilities. It supports users in flexibly defining indicator content according to specific scenarios, freely designing the hierarchical structure of indicators, and dynamically adjusting the composition of the indicator system based on changing needs. Please refer to... Figure 1 The system mainly includes an assessment standard management module, an assessment data management module, an assessment model management module, an assessment plug-in management module, an assessment system management module, and an assessment scheme management module. The following section will discuss these modules in conjunction with... Figure 1 The specific functions and implementation of each module are explained in detail.

[0024] The evaluation specification management module primarily stores evaluation plugin integration specifications, evaluation data description specifications, evaluation plugin development specifications, and evaluation model description specifications. These evaluation specifications are formed based on the organization and analysis of evaluation task documents, evaluation systems, and evaluation indicators. In this embodiment's dynamic multi-level evaluation system, evaluation plugin integration specifications, evaluation data description specifications, evaluation plugin development specifications, and evaluation model description specifications are defined. These specifications provide guidelines for the integration of evaluation plugins, the definition and design of evaluation data, the development of evaluation plugins, and the design of evaluation models.

[0025] Furthermore, the evaluation data management module provides two core functions: evaluation data model creation and data preprocessing development. The evaluation data model creation function creates the required evaluation data based on the actual task requirements and predefined evaluation data description specifications. Once created, the evaluation data model can acquire the corresponding data according to the actual task requirements. Additionally, an evaluation scheme may require preprocessing of the input data. This evaluation data management module also provides the development of a data preprocessing model. The data preprocessing model is used to preprocess the input data according to the requirements of the evaluation scheme, and the preprocessed data can be used as the input data for the evaluation model.

[0026] Please continue to refer to this. Figure 1 The evaluation model management module primarily includes evaluation models developed based on the evaluation model description specification, supporting the direct development and debugging of evaluation models within a dynamic multi-level evaluation system. In practical applications, evaluation model components can be developed according to the specification and integrated into the dynamic multi-level evaluation system through the model registration function. The developed evaluation model can run as an independent model or be transformed into a drag-and-drop component to participate in the design of more complex, comprehensive computational models that better meet actual business needs. This comprehensive computational model can also call designed evaluation plugins.

[0027] The evaluation plugin management module primarily develops and integrates evaluation plugins based on the evaluation plugin development specifications, ensuring system consistency and compatibility. The dynamic multi-level evaluation system boasts high scalability. Third-party developers only need to develop data model plugins, evaluation model plugins, and evaluation indicator system plugins according to relevant specifications, encapsulate them according to the evaluation integration specifications, and register the developed evaluation plugins into the system through system registration. This flexible expansion method provides strong support for system functional expansion and adaptation to different business needs.

[0028] Please continue to refer to this. Figure 1 The evaluation system management module is used to design and manage the multi-level indicator system required for evaluation tasks and to associate it with the evaluation model. In this embodiment, it mainly consists of three parts: final-level indicator design, comprehensive model calculation binding, and indicator system design. In the indicator system design process, the hierarchical structure of the indicator system must first be clarified, and then the specific indicators included in each level must be determined. Indicators are divided into final-level indicators and comprehensive calculation indicators according to calculation rules. Final-level indicators are associated with the defined evaluation data model and evaluation model, and are required to be measurable and independent; other indicators besides final-level indicators are called comprehensive calculation indicators, which need to be bound to the comprehensive calculation model.

[0029] The evaluation scheme management module creates evaluation schemes based on a multi-level indicator system selected from the evaluation tasks. Executing these evaluation schemes generates evaluation results. In this embodiment, each evaluation scheme is directly associated with a specific evaluation task. The indicator system is a general resource with fully configured data and models, while the evaluation scheme is a specific implementation plan for applying these indicator systems to actual business scenarios. When creating an evaluation scheme, users can choose a pre-built evaluation system as a template; if the existing indicator system cannot meet business needs, the system also supports creating a new evaluation indicator system from scratch. After the evaluation scheme is built, the evaluation task can be executed to obtain the final evaluation results.

[0030] To further illustrate the evaluation process of the dynamic multi-level evaluation system proposed in this embodiment, the following description uses the execution process of an evaluation scheme. Please refer to... Figure 2 Specifically, it includes: S101. Obtain evaluation plan information and generate execution diagram.

[0031] When preparing to execute an evaluation plan, the system extracts the evaluation plan information and generates an execution graph based on it. To avoid evaluation plan execution failure due to entering an infinite loop, after generating the execution graph, it is necessary to determine whether an infinite loop exists in the graph. If it does, the system displays an error message; otherwise, it executes subsequent operations.

[0032] S102. Convert the nodes in the execution graph into an execution sequence, and execute the nodes in the sequence in sequence.

[0033] Please continue to refer to this. Figure 2 The execution process of any single node will be described, including: S1021. Obtain the evaluation model information in the node and determine whether it is a comprehensive calculation model. If not, proceed to S1022; if yes, proceed to S1025.

[0034] S1022. Obtain input data.

[0035] In this embodiment, when acquiring input data, the data source can be determined by evaluating the data model to identify whether the data originates from a file, a database, or an API interface. If the data originates from a file, it is retrieved from the file storage; if the data originates from a database, it is retrieved from the database; otherwise, the data is retrieved through the API interface.

[0036] S1023. Determine whether preprocessing is required.

[0037] After obtaining the input data, if data preprocessing is required, the data preprocessing model is executed; otherwise, proceed directly to S1024. S1024. Execute the corresponding evaluation model.

[0038] When executing the evaluation model, it is necessary to determine whether the model originates from an external plugin. If the model originates from an external plugin, the evaluation plugin is executed; otherwise, the evaluation model is executed directly. After execution, upstream result data is generated and saved to the database.

[0039] S1025. Obtain upstream result data from the database and calculate the comprehensive calculation index results.

[0040] S103. Save the calculation results of each node and output the evaluation results.

[0041] In one embodiment, please refer to... Figure 2 After obtaining the evaluation results, the process also includes S104, which determines whether the evaluation scheme is configured for periodic execution. If so, the process of S101 to S103 is repeated according to the configured period. If not, it will further determine whether the scheme requires real-time execution. If it is real-time execution, the current evaluation scheme will be executed cyclically. Otherwise, the current evaluation scheme will only be executed once.

[0042] Based on existing methodologies and frameworks for indicator systems, this invention develops a flexible and practical system for constructing and applying evaluation systems, enabling the effective integration of data, models, and indicator systems. The system is logically clear, easy to operate, and significantly improves the efficiency of evaluation work. This invention mainly has the following advantages: (1) Scenario adaptability. Through the design of a multi-level indicator system and the integration mechanism of quantitative and qualitative indicators, the system can flexibly adapt to the assessment needs of different fields such as finance, energy, and healthcare.

[0043] (2) Dynamic Adjustment Capability. The system supports the free splitting and recombination of evaluation indicators, allowing for flexible selection of required indicators based on specific business needs, and real-time adjustment and optimization of existing indicators. Furthermore, this method allows for dynamic correction of indicator parameters, ensuring the accuracy and adaptability of the evaluation. The built-in expert scoring mechanism can be used flexibly according to actual needs; the system also supports hybrid evaluation models, enabling both data-driven automatic evaluation and comprehensive evaluation combining expert scoring. Simultaneously, it provides a dynamic adjustment function for expert weights, allowing for flexible adjustment of the level of expert participation based on different evaluation scenarios, further enhancing the flexibility and accuracy of the evaluation. (3) Flexibility and Scalability. The multi-dimensional and multi-level indicator system constructed by this system can flexibly adjust the weights, data sources, and evaluation algorithms of indicators according to different business needs and application scenarios. At the same time, the flexible design of the system allows for the addition, deletion, and modification of indicators, ensuring that the indicator system can be dynamically updated with business development and technological progress. In addition, through defined specifications, open interfaces, and standardized data formats, the system supports seamless integration with other systems and tools, and can be easily extended to more complex analysis scenarios, such as the introduction of real-time computing, big data analysis, and machine learning models. This flexibility and scalability not only ensures the applicability of the indicator system in the current scenario, but also provides a solid foundation for future optimization and upgrades.

[0044] It should be noted that, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances. The accompanying drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0045] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A dynamic multi-level evaluation system, characterized in that, include: The assessment specification management module is used to store assessment plugin integration specifications, assessment data description specifications, assessment plugin development specifications, and assessment model description specifications. The assessment data management module creates the assessment data model required for the task based on the assessment data description specification. The evaluation model management module develops evaluation models based on the evaluation model description specification and calls evaluation plugins to complete the design of the evaluation models. The evaluation plugin management module is used to manage evaluation plugins developed based on the evaluation plugin development specification. The evaluation plugins are encapsulated according to the evaluation plugin integration specification and are called by the evaluation model management module. The assessment system management module is used to design and manage the multi-level indicator system required for assessment tasks and to link it with the assessment model; The evaluation scheme management module allows you to create an evaluation scheme by selecting a multi-level indicator system based on the evaluation task. Executing the evaluation scheme will generate the evaluation results.

2. The dynamic multi-level evaluation system according to claim 1, characterized in that, The evaluation plugin integration specification, evaluation data description specification, evaluation plugin development specification, and evaluation model description specification are all based on the collation and analysis of evaluation task documents, evaluation system, and evaluation indicators.

3. The dynamic multi-level evaluation system according to claim 1, characterized in that, The evaluation data management module also includes a preprocessing model for preprocessing the evaluation data.

4. The dynamic multi-level evaluation system according to claim 1, characterized in that, The evaluation model management module includes an evaluation model developed based on the evaluation data model and a comprehensive calculation model designed by combining the evaluation model and the evaluation query plugin.

5. The dynamic multi-level evaluation system according to claim 1, characterized in that, The design of the multi-level indicator system includes: determining the hierarchical structure of the indicator system, and determining the specific indicators included in each level. The indicators are divided into final-level indicators and comprehensive calculation indicators. The final-level indicators are associated with the evaluation data model and the evaluation model. The comprehensive calculation indicators are bound to the designed comprehensive calculation model.

6. The dynamic multi-level evaluation system according to claim 1, characterized in that, In the evaluation scheme management module, the execution process of the evaluation scheme includes: Obtain evaluation plan information and generate an execution diagram; Convert the nodes in the execution graph into execution sequences, and execute the nodes in the sequences sequentially; Save the calculation results of each node and output the evaluation results.

7. The dynamic multi-level evaluation system according to claim 6, characterized in that, The specific execution process of any node in the execution sequence includes: Obtain the evaluation model information from the nodes and determine whether it is a comprehensive calculation model; If it is not a comprehensive calculation model, the input data is obtained, and the data preprocessing model is called to preprocess the data according to business needs, or the evaluation model is executed directly, and the upstream result data generated by the execution is saved to the database; when executing the evaluation model, if the evaluation model comes from an external plugin, the corresponding evaluation plugin is executed directly, otherwise the evaluation model is executed directly. If it is a comprehensive calculation model, then the upstream result data is obtained, and the comprehensive calculation index result is calculated.

8. The dynamic multi-level evaluation system according to claim 6, characterized in that, After saving the calculation results of each node, it also includes: Determine whether the evaluation plan is configured for periodic execution; If the configuration period is executed, then according to the configured period, the nodes in the sequence will be re-evaluated sequentially. If periodic execution is not configured, it further determines whether real-time execution is required. If real-time execution is required, the current evaluation scheme is executed repeatedly; otherwise, the current evaluation scheme is executed only once.

9. The dynamic multi-level evaluation system according to claim 7, characterized in that, The acquisition of input data specifically includes: The data model is evaluated to determine whether the source of the input data is a file, a database, or an API interface. If the data is determined to be from a file, it is retrieved from the file storage; if the data is determined to be from a database, it is retrieved from the database; otherwise, the data is retrieved through the API interface.

10. The dynamic multi-level evaluation system according to claim 6, characterized in that, After generating the execution graph, check if there are infinite loops in the execution graph. If so, display an error message. If it does not exist, proceed with the subsequent operations.

Images