Multi-level application evaluation system and method thereof

Abstract

The invention discloses a multi-level application evaluation system and method, and relates to the technical field of application evaluation, and the system comprises a business layer, an evaluation basic platform layer and a support layer. The evaluation basic platform layer comprises a basic subsystem, a core module and a professional database. On the basis of the evaluation operation basic platform, corresponding evaluation application environments are provided for different evaluation requirements of different evaluation contents, different evaluation data, different evaluation methods and different evaluation styles, and the evaluation application environments are used for supporting customized evaluation requirements of a command training system and a simulation system.

Description

Technical Field

[0001] The present invention relates to the technical field of application evaluation, and in particular to a multi-level application evaluation system and method thereof. Background Art

[0002] Traditional evaluation systems only provide a single evaluation calculation process and model. This method cannot meet the needs when the evaluation content, data sources and evaluation styles are diversified.

[0003] In the field of evaluation, for the evaluation of a single task, the usual business process is: experts select indicators, then build an indicator system based on the indicator set, and then create an evaluation plan (including the creation and selection of evaluation algorithms), collect data, create evaluation tasks, execute evaluation tasks, and then obtain evaluation results and generate evaluation reports. This method cannot meet the needs when the evaluation content, data sources, and evaluation styles are diversified. For complex evaluation applications, there is an urgent need for a multi-level application evaluation system and its method, which is based on the evaluation operation basic platform and is designed to form a professional evaluation application environment for differentiated evaluation needs of different evaluation contents, different evaluation data, different evaluation methods, and different evaluation styles to support the customized evaluation needs of command training systems and simulation systems. Summary of the invention

[0004] The object of the present invention is to provide a multi-level application evaluation system and method thereof to improve the above technical problems.

[0005] In order to achieve the above-mentioned object of the invention, the embodiment of the present invention provides the following technical solutions:

[0006] A multi-level application evaluation system, comprising a business layer, an evaluation basic platform layer and a support layer; the evaluation basic platform layer comprises a basic subsystem, a core module and a professional database;

[0007] The business layer is used to provide an application operating environment for evaluating the basic platform layer; the application operating environment includes an evaluation application operating environment based on simulation data analysis and an evaluation application operating environment based on expert discussion;

[0008] The basic subsystem is used to determine the application type of the multi-level application evaluation system, and to design and manage the corresponding evaluation indicators, evaluation operators, evaluation models and evaluation tasks, and to call the core module to perform evaluation and generate a corresponding evaluation result report; the application types include command process evaluation, effect-based combat plan evaluation and command confrontation training evaluation;

[0009] The core module is used to store evaluation operators and evaluation models, and provide visualization components, evaluation calculation engines, report generation engines, and data visualization engines for the basic subsystem;

[0010] The professional database is used to store the basic subsystem and the data in the basic subsystem;

[0011] The support layer is used to provide hardware equipment and software equipment to support the normal operation of the multi-level application evaluation system.

[0012] Furthermore, the basic subsystem includes:

[0013] The data access and preprocessing subsystem is used to obtain various types of simulation data, training data, solution data, and expert evaluation data and perform preprocessing; the preprocessing includes formatting, parsing, cleaning, classification, and extraction;

[0014] The evaluation task management subsystem is used to determine the application type of the multi-level application evaluation system; the application type includes evaluation purpose, evaluation background, evaluation subject, evaluation object, evaluation content, and evaluation data source;

[0015] An evaluation indicator design and management subsystem, which is used to design or call corresponding indicators for evaluation of different application operating environments and different application types based on the application type, and establish a corresponding indicator system based on the called indicators;

[0016] An evaluation operator design and management subsystem, used to provide corresponding algorithms for evaluation of different application operating environments and different application types based on the application type;

[0017] The evaluation model design and management subsystem is used to provide corresponding evaluation models for evaluations of different application operating environments and different application types; the evaluation model is an evaluation scheme, that is, multiple algorithms selected according to the evaluations of different application operating environments and different application types;

[0018] An evaluation process execution subsystem is used to call the evaluation calculation engine to run the evaluation model and obtain an evaluation result;

[0019] The evaluation result display and report generation subsystem is used to call the visualization component, the report generation engine and the data visualization engine to generate and display a visualization evaluation report.

[0020] Furthermore, the core module includes:

[0021] Evaluation operator library, used to store algorithms corresponding to different application operating environments and different application types;

[0022] Evaluation model library, used to store evaluation plans corresponding to different application operating environments and different application types;

[0023] Visual component library, used to provide visualization functions;

[0024] Evaluation calculation engine, used to calculate evaluation results for different application running environments and different application types;

[0025] Report generation engine, used to generate visual assessment reports;

[0026] Data visualization engine for visualizing multi-tier application evaluation systems.

[0027] Furthermore, the professional database includes:

[0028] Evaluation criteria library, used to formulate and store evaluation criteria;

[0029] Report template library, which provides different formats for visual assessment reports;

[0030] Metadatabase, used to store various simulation data, training data, solution data, and expert evaluation data;

[0031] Expert database, used to store basic information of different scoring experts and set corresponding expert authority; the basic information of the scoring experts includes user name, password, profession and unit;

[0032] Evaluation indicator library, used to store evaluation indicators;

[0033] The evaluation indicator system library is used to store the evaluation indicator system.

[0034] A multi-level application evaluation method, comprising:

[0035] S1. Determine an application type through the basic subsystem; based on the application type, provide a corresponding application operating environment through the application layer;

[0036] S2. Determine hardware equipment and software equipment through the support layer;

[0037] S3. Based on the application type, the data to be evaluated is obtained through the basic subsystem; the data to be evaluated is processed through the core module and the data professional library, and a corresponding visual evaluation result report is generated and displayed.

[0038] Furthermore, the basic subsystem includes a data access and preprocessing subsystem, an evaluation task management subsystem, an evaluation indicator design and management subsystem, an evaluation operator design and management subsystem, an evaluation model design and management subsystem, an evaluation process execution subsystem, and an evaluation result display and report generation subsystem;

[0039] The core modules include an evaluation operator library, an evaluation model library, a visualization component library, an evaluation calculation engine, a report generation engine, and a data visualization engine;

[0040] The professional database includes an evaluation standard library, a report template library, a metadata library, an evaluation index library and an evaluation index system library.

[0041] Furthermore, based on the application type, the data to be evaluated is obtained through the basic subsystem; the data to be evaluated is processed through the core module and the data professional library, and a corresponding visual evaluation result report is generated and displayed, including:

[0042] S3-1, obtaining all corresponding evaluation data through the data access and preprocessing subsystem, preprocessing them, and transmitting them to the metadata database for storage;

[0043] S3-2, selecting experts through the expert database;

[0044] S3-3, selecting the preprocessed evaluation data through the evaluation index design and management subsystem and the selected experts to obtain corresponding evaluation index data, and transmitting the data to the evaluation index library for storage;

[0045] S3-4, based on the evaluation indicator data, formulate corresponding evaluation standards through the evaluation indicator design and management subsystem, and transmit to the evaluation standard library for storage;

[0046] S3-5, based on the evaluation criteria, construct a corresponding evaluation indicator system through the evaluation indicator design and management subsystem, and transmit it to the evaluation indicator system library for storage;

[0047] S3-6, determining the application type through the assessment task management subsystem;

[0048] S3-7, based on the application type, calling the corresponding evaluation indicator system in the evaluation indicator system through the evaluation indicator design and management subsystem;

[0049] S3-8, based on the application type and the called evaluation indicator system, calling the algorithm in the evaluation operator library through the evaluation operator design and management subsystem;

[0050] S3-9, determining the order or process of the called algorithms through the evaluation model design and management subsystem, obtaining the corresponding evaluation model, and transmitting it to the evaluation model library for storage; the evaluation model is an evaluation plan;

[0051] S3-10, calling the evaluation calculation engine through the evaluation process execution subsystem to execute the evaluation plan and obtain the corresponding evaluation result;

[0052] S3-11. Based on the application type, select a corresponding report template from the report templates;

[0053] S3-12. Based on the selected report template, the evaluation result display and report generation subsystem calls the visualization component, the report generation engine and the data visualization engine to visualize the evaluation result, generate a visual evaluation report and display it.

[0054] Further, in S3-7, the application type is matched with the evaluation indicator system in the evaluation indicator system. If the matching result is: there is a corresponding evaluation indicator system, then enter S3-8; if the matching result is: there is no corresponding evaluation indicator system, then enter the following steps:

[0055] Based on the application type, corresponding data to be evaluated is obtained, corresponding evaluation indicators, evaluation standards and evaluation indicator systems are formulated through the evaluation indicator design and management subsystem, and transmitted to the evaluation indicator library, the evaluation standard library and the evaluation indicator system library respectively;

[0056] Based on the application type and the called evaluation index system, the algorithm in the evaluation operator library is called through the evaluation operator design and management subsystem; the order or process of the called algorithm is determined through the evaluation model design and management subsystem, the corresponding evaluation model is obtained, and it is transmitted to the evaluation model library for storage; enter S3-10.

[0057] The beneficial effects of the present invention are as follows: the present invention is based on the evaluation operation basic platform, and provides a corresponding evaluation application environment for differentiated evaluation needs of different evaluation contents, different evaluation data, different evaluation methods, and different evaluation styles, so as to support customized evaluation needs of command training systems and simulation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 is a system structure diagram in an embodiment of the present invention;

[0060] Figure 2 is a flow chart of a method in an embodiment of the present invention;

[0061] Figure 3 A flow chart of a method for model-based evaluation of an application operating environment in an embodiment of the present invention;

[0062] Figure 4A method flow chart of an application operating environment based on simulation data analysis in an embodiment of the present invention;

[0063] Figure 5 The flowchart is a method for evaluating an application operating environment based on expert discussion in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0065] Embodiment 1:

[0066] See also Figure 1 , a multi-level application evaluation system provided in this embodiment includes a business layer, an evaluation basic platform layer and a support layer; the evaluation basic platform layer includes a basic subsystem, a core module and a professional database;

[0067] The business layer is used to provide an application operating environment for evaluating the basic platform layer; the application operating environment includes an evaluation application operating environment based on simulation data analysis and an evaluation application operating environment based on expert discussion, and also includes an application operating environment based on model evaluation;

[0068] The evaluation application operating environment based on simulation data analysis is an application environment for analyzing the strengths and weaknesses of a single combat plan, and evaluating the feasibility, effectiveness, and risk of the operation;

[0069] The evaluation application operating environment based on expert discussion is an application environment based on expert discussion that uses expert groups to evaluate subjective problems that cannot be modeled.

[0070] The application operating environment including the model-based evaluation is an application environment for evaluating structured evaluation problems using evaluation models.

[0071] The basic subsystem is used to determine the application type of the multi-level application evaluation system, and to design and manage the corresponding evaluation indicators, evaluation operators, evaluation models and evaluation tasks, and to call the core module to perform evaluation and generate a corresponding evaluation result report; the application types include command process evaluation, effect-based combat plan evaluation and command confrontation training evaluation;

[0072] The basic subsystem includes:

[0073] The data access and preprocessing subsystem is used to obtain various types of simulation data, training data, solution data, and expert evaluation data and perform preprocessing; the preprocessing includes formatting, parsing, cleaning, classification, and extraction;

[0074] The evaluation task management subsystem is used to determine the application type of the multi-level application evaluation system; the application type includes evaluation purpose, evaluation background, evaluation subject, evaluation object, evaluation content, and evaluation data source; the evaluation object is at least one;

[0075] An evaluation indicator design and management subsystem, which is used to design or call corresponding indicators for evaluation of different application operating environments and different application types based on the application type, and establish a corresponding indicator system based on the called indicators;

[0076] An evaluation operator design and management subsystem, used to provide corresponding algorithms for evaluation of different application operating environments and different application types based on the application type;

[0077] The evaluation model design and management subsystem is used to provide corresponding evaluation models for evaluations of different application operating environments and different application types; the evaluation model is an evaluation scheme, that is, multiple algorithms selected according to the evaluations of different application operating environments and different application types;

[0078] An evaluation process execution subsystem is used to call the evaluation calculation engine to run the evaluation model and obtain an evaluation result;

[0079] The evaluation result display and report generation subsystem is used to call the visualization component, the report generation engine and the data visualization engine to generate and display a visualization evaluation report.

[0080] The core module is used to store evaluation operators and evaluation models, and provide visualization components, evaluation calculation engines, report generation engines, and data visualization engines for the basic subsystem;

[0081] The professional database is used to store the basic subsystem and the data in the basic subsystem;

[0082] The professional database includes:

[0083] Evaluation criteria library, used to formulate and store evaluation criteria;

[0084] Report template library, which provides different formats for visual assessment reports;

[0085] Metadatabase, used to store various simulation data, training data, solution data, and expert evaluation data;

[0086] Expert database, used to store basic information of different scoring experts and set corresponding expert authority; the basic information of the scoring experts includes user name, password, profession and unit;

[0087] Evaluation indicator library, used to store evaluation indicators;

[0088] The evaluation indicator system library is used to store the evaluation indicator system.

[0089] The core modules include:

[0090] Evaluation operator library, used to store algorithms corresponding to different application operating environments and different application types;

[0091] Evaluation model library, used to store evaluation plans corresponding to different application operating environments and different application types;

[0092] Visual component library, used to provide visualization functions;

[0093] Evaluation calculation engine, used to calculate evaluation results for different application running environments and different application types;

[0094] Report generation engine, used to generate visual assessment reports;

[0095] Data visualization engine for visualizing multi-tier application evaluation systems.

[0096] The support layer is used to provide hardware and software equipment to support the normal operation of the multi-level application evaluation system. The hardware equipment is seats, servers (web, database); the software equipment includes storage backup system and operating system; the storage backup system includes third-party office software Office2007 or above (any one of DAMO, SHENTO, MongDB, Oracle, SQLServer and Sqlite), WPS third-party computing software (R, Matlab, Python); the operating system can be any one of 32-bit or 64-bit WindowXP, Win7, Win10, Linux, Kylin.

[0097] The multi-level application evaluation system adopts the service-oriented (SOA) design concept, object-oriented development method and extensible open system architecture to integrate the established application system and development tools horizontally and vertically. Through the layered architecture, it provides standardized interfaces between layers and establishes a system platform to provide strong support for the expansion of the multi-level application evaluation system.

[0098] The present invention sets up multi-layer modules and refines the function allocation so that each functional module performs its own duties and is interconnected, which reduces the calculation difficulty and processing speed to a certain extent. The system is fully functional and covers a wide range, and can be changed according to different application types, such as evaluation applications based on simulation data analysis, evaluation applications based on expert discussion and analysis, etc., with high applicability and pertinence.

[0099] like Figure 2 As shown, a multi-level application evaluation method includes:

[0100] S1. Determine the application type through the basic subsystem; based on the application type, provide a corresponding application operating environment through the application layer; the current application operating environment is an application operating environment based on model evaluation.

[0101] S2. Determine hardware equipment and software equipment through the support layer;

[0102] S3. Based on the application type, the data to be evaluated is obtained through the basic subsystem; the data to be evaluated is processed through the core module and the data professional library, and a corresponding visual evaluation result report is generated and displayed.

[0103] The basic subsystem includes a data access and preprocessing subsystem, an evaluation task management subsystem, an evaluation indicator design and management subsystem, an evaluation operator design and management subsystem, an evaluation model design and management subsystem, an evaluation process execution subsystem, and an evaluation result display and report generation subsystem;

[0104] The core modules include an evaluation operator library, an evaluation model library, a visualization component library, an evaluation calculation engine, a report generation engine, and a data visualization engine;

[0105] The professional database includes an evaluation standard library, a report template library, a metadata library, an evaluation index library and an evaluation index system library.

[0106] like Figure 3 As shown, based on the application type, the data to be evaluated is obtained through the basic subsystem; the data to be evaluated is processed through the core module and the data professional library, and a corresponding visual evaluation result report is generated and displayed, including:

[0107] S3-1. Obtain all corresponding evaluation data through the data access and preprocessing subsystem and perform preprocessing, and transmit to the metadata database for storage; the evaluation data includes various simulation data, training data, solution data, expert evaluation data, that is, various subjective and objective data, and are all compatible with various databases such as Oracle and MySQL.

[0108] S3-2, selecting experts through the expert database;

[0109] S3-3, the pre-processed evaluation data is selected by the evaluation indicator design and management subsystem and the selected experts to obtain the corresponding evaluation indicator data, and transmit it to the evaluation indicator library for storage; the evaluation indicator data includes effectiveness indicators and performance indicators; the effectiveness indicator is an indicator that depends on other indicators for calculation; wherein the dependent indicator is called a sub-indicator, and the indicator that depends on other indicators is called a parent indicator. The performance indicator is an indicator that directly measures the specific data and directly observes the event to determine the completion of the task; wherein the performance indicator does not depend on other indicators.

[0110] S3-4. Based on the evaluation indicator data, the corresponding evaluation standards are formulated through the evaluation indicator design and management subsystem, and transmitted to the evaluation standard library for storage; the level, grade or threshold that the evaluation indicator data needs to reach is clarified, such as operation time. If the operation time is less than 5 minutes, the corresponding score is excellent, and if the operation time is greater than 5 minutes, the corresponding score is low.

[0111] S3-5, based on the evaluation criteria, construct a corresponding evaluation indicator system through the evaluation indicator design and management subsystem, and transmit it to the evaluation indicator system library for storage;

[0112] S3-6, determining the application type through the assessment task management subsystem;

[0113] S3-7, based on the application type, calling the corresponding evaluation indicator system in the evaluation indicator system through the evaluation indicator design and management subsystem;

[0114] In S3-7, the application type is matched with the evaluation indicator system in the evaluation indicator system. If the matching result is: there is a corresponding evaluation indicator system, then enter S3-8; if the matching result is: there is no corresponding evaluation indicator system, then enter the following steps:

[0115] Based on the application type, corresponding data to be evaluated is obtained, corresponding evaluation indicators, evaluation standards and evaluation indicator systems are formulated through the evaluation indicator design and management subsystem, and transmitted to the evaluation indicator library, the evaluation standard library and the evaluation indicator system library respectively;

[0116] Based on the application type and the called evaluation index system, the algorithm in the evaluation operator library is called through the evaluation operator design and management subsystem; the order or process of the called algorithm is determined through the evaluation model design and management subsystem, the corresponding evaluation model is obtained, and it is transmitted to the evaluation model library for storage; enter S3-10.

[0117] The algorithms include calculation of mean, variance, covariance, correlation coefficient, Pearson correlation coefficient and skewness, kurtosis, principal component analysis, analytic hierarchy process, fuzzy synthesis, TOPSIS, data envelopment method, grey whitening weight function and ADC, as well as other existing algorithms. According to actual needs or application types, the evaluation algorithm can be extended by using script operators, formula operators or DLL operators, and the extended evaluation algorithms are stored in the evaluation operator library. The evaluation model is an evaluation plan, including multiple algorithms that determine the process or sequence. The evaluation plan is the basis for evaluating the same type of evaluation objects.

[0118] S3-8, based on the application type and the called evaluation indicator system, calling the algorithm in the evaluation operator library through the evaluation operator design and management subsystem;

[0119] S3-9, determining the order or process of the called algorithms through the evaluation model design and management subsystem, obtaining the corresponding evaluation model, and transmitting it to the evaluation model library for storage; the evaluation model is an evaluation plan;

[0120] S3-10, calling the evaluation calculation engine through the evaluation process execution subsystem to execute the evaluation plan and obtain the corresponding evaluation result;

[0121] S3-11. Based on the application type, select a corresponding report template from the report templates;

[0122] S3-12. Based on the selected report template, the evaluation result display and report generation subsystem calls the visualization component, the report generation engine and the data visualization engine to visualize the evaluation result, generate a visual evaluation report and display it. The visual evaluation report includes the corresponding indicator system, evaluation plan, evaluation results and other information. The evaluation report uses text and charts to display various types of information.

[0123] Embodiment 2:

[0124] like Figure 4 As shown, a multi-level application evaluation method for evaluating an application for simulation data analysis, wherein the application operating environment is an evaluation application operating environment based on simulation data analysis, comprising:

[0125] A1. Select the corresponding combat plan based on combat requirements; perform architecture modeling and design on the simulation system to obtain the corresponding system model; perform scenario editing and assumptions on the system model, and verify the system model through 3D preview;

[0126] A2. Conduct experimental design for entity attributes in the verified system model based on simulation scenarios, generate corresponding simulation test plans, and use them as experimental samples; simulation test plans include initial single combat plans and initial multiple combat plans;

[0127] A3. Set simulation parameters, collect simulation data and process them to obtain corresponding simulation result data;

[0128] A3 further comprises the following steps:

[0129] A3-1. Set simulation parameters and use them as initial excitation conditions to drive simulation execution; simulation parameters include basic target parameters, target equivalent model, warhead structure model, basic warhead parameters, strike scenario parameters, and experimental plan;

[0130] A3-2. Apply the simulation test plan to the verified system model and use it to collect simulation data, that is, deploy simulation members, execute the simulation process, monitor the simulation process, and collect simulation return data and simulation result data corresponding to the simulation record; the simulation result data is evolved from the simulation experiment plan, and the simulation result data includes a single operation plan and multiple operation plans;

[0131] A4. Evaluate the simulation result data through a multi-level application evaluation system to obtain a corresponding visual evaluation report;

[0132] A4 further comprises the following steps:

[0133] A4-1. Determine whether the simulation result data is multiple combat plans. If so, proceed to A4-2; otherwise, proceed to A4-3.

[0134] A4-2. Process multiple combat plans through a multi-level application evaluation system to obtain corresponding visual evaluation reports;

[0135] A4-2 further comprises the following steps:

[0136] A4-2-1. Multiple operational plans are adopted and preprocessed through the data access and preprocessing subsystem, and then transmitted to the metadata database for storage; preprocessing includes data segmentation, data filtering, classification and aggregation, attribute compression and transformation, attribute filtering, attribute value filling and attribute calculation;

[0137] A4-2-2. Select experts through the expert database;

[0138] A4-2-3. Based on operational requirements, the evaluation index design and management subsystem and selected experts select the pre-processed multiple operational plans to obtain corresponding simulation evaluation index data, and transmit them to the evaluation index library for storage;

[0139] A4-2-4. Based on the simulation evaluation index data, the corresponding simulation evaluation standards are formulated through the evaluation index design and management subsystem, and transmitted to the evaluation standard library for storage;

[0140] A4-2-5. Based on the simulation evaluation standard, the corresponding simulation evaluation index system is constructed through the evaluation index design and management subsystem, and transmitted to the evaluation index system library for storage;

[0141] A4-2-6. Based on the simulation evaluation index system, the algorithms in the evaluation operator library are called through the evaluation operator design and management subsystem;

[0142] A4-2-7. Determine the order or process of the algorithm to be called through the evaluation model design and management subsystem, obtain the corresponding simulation evaluation model, and transfer it to the evaluation model library for storage; the simulation evaluation model is a sensitivity analysis model and a pros and cons comparison evaluation;

[0143] A4-2-9. The evaluation process execution subsystem calls the evaluation calculation engine to execute the simulation evaluation model to process the simulation evaluation index system and obtain the corresponding simulation evaluation results;

[0144] A4-2-10. Select the report template that corresponds to the simulation data.

[0145] A4-2-11. Based on the selected report template, the simulation evaluation results are visualized by calling the visualization component, report generation engine and data visualization engine through the evaluation result display and report generation subsystem, and a visual simulation evaluation report is generated and displayed.

[0146] A4-3. Process a single combat plan through a multi-level application evaluation system to obtain a corresponding visual evaluation report.

[0147] A4-3 further comprises the following steps:

[0148] A4-3-1. A single operation plan is adopted and preprocessed through the data access and preprocessing subsystem, and then transmitted to the metadata database for storage; preprocessing includes data segmentation, data filtering, classification and aggregation, attribute compression and transformation, attribute filtering, attribute value filling and attribute calculation;

[0149] A4-3-2. Select experts through the expert database;

[0150] A4-3-3. Based on operational requirements, the pre-processed single operational plan is selected by the evaluation index design and management subsystem and selected experts to obtain corresponding simulation evaluation index data, and the data is transmitted to the evaluation index library for storage;

[0151] A4-3-4. Based on the simulation evaluation index data, the corresponding simulation evaluation standards are formulated through the evaluation index design and management subsystem, and transmitted to the evaluation standard library for storage;

[0152] A4-3-5. Based on the simulation evaluation standard, the corresponding simulation evaluation index system is constructed through the evaluation index design and management subsystem, and transmitted to the evaluation index system library for storage;

[0153] A4-3-6. Based on the simulation evaluation index system, the algorithms in the evaluation operator library are called through the evaluation operator design and management subsystem;

[0154] A4-3-7. Determine the order or process of the algorithm to be called through the evaluation model design and management subsystem, obtain the corresponding simulation evaluation model, and transfer it to the evaluation model library for storage; the simulation evaluation model is a feasibility analysis model, a benefit analysis model, and a risk analysis model;

[0155] A4-3-9. The evaluation process execution subsystem calls the evaluation calculation engine to execute the simulation evaluation model to process the simulation evaluation index system and obtain the corresponding simulation evaluation results;

[0156] A4-3-10. Select the report template that corresponds to the simulation data.

[0157] A4-3-11. Based on the selected report template, the simulation evaluation results are visualized by calling the visualization component, report generation engine and data visualization engine through the evaluation result display and report generation subsystem, and a visual simulation evaluation report is generated and displayed.

[0158] Embodiment 3:

[0159] like Figure 5 As shown, a multi-level application evaluation method for an evaluation application for expert discussion and analysis, wherein the application operating environment is an evaluation application operating environment based on expert discussion, including:

[0160] B1. Select experts through the expert database; select expert discussion evaluation data (expert evaluation data) through the evaluation index design and management subsystem and the selected experts to obtain corresponding expert discussion evaluation index data, and transmit them to the evaluation index database for storage; in addition, coherence analysis and principal component analysis can be performed on the expert discussion evaluation index data, and the expert discussion evaluation index data that meets the preset threshold conditions are selected as the final expert discussion evaluation index data;

[0161] B2. Based on the expert discussion evaluation index data, the corresponding expert discussion evaluation standard is formulated through the evaluation index design and management subsystem, and transmitted to the evaluation standard library for storage;

[0162] B3. Based on the expert discussion evaluation standards, the corresponding expert discussion evaluation indicator system is constructed through the evaluation indicator design and management subsystem, the hierarchy is set for the indicators and the dependency relationship between the indicators is established, and the indicators that are not suitable for direct measurement are decomposed to establish an expert discussion indicator system consisting of effectiveness indicators (MOE) and performance indicators (MOP), and the system is transmitted to the evaluation indicator system library for storage;

[0163] B4. Manage the expert database and set expert scoring rules;

[0164] Add the corresponding scoring experts to the expert database; set the basic information of the expert, including user name, password, major, unit and other basic information, that is, edit the expert information; set the authority of the expert: set the authority of the expert;

[0165] Setting expert scoring rules for the expert discussion and evaluation indicator system; the scoring rules include scoring the expert discussion and evaluation indicators by value or grade, as well as the scoring experts and the degree of familiarity of the experts;

[0166] B5. Determine the order or process of the algorithm to be called through the evaluation model design and management subsystem, obtain the corresponding expert discussion evaluation model, and transmit it to the evaluation model library for storage;

[0167] B6. Determine the application type corresponding to the expert discussion through the evaluation task management subsystem; based on the application type corresponding to the expert discussion, call the corresponding expert discussion evaluation index system in the evaluation index system through the evaluation index design and management subsystem;

[0168] B7. Based on the application type corresponding to the expert discussion and the expert discussion evaluation index system called, the algorithm in the evaluation operator library is called through the evaluation operator design and management subsystem;

[0169] B8. Determine the order or process of the algorithm to be called through the evaluation model design and management subsystem, obtain the corresponding expert discussion evaluation model, and transmit it to the evaluation model library for storage;

[0170] B9. Create evaluation tasks based on the expert discussion evaluation model; assign evaluation tasks to scoring experts in the expert database according to the scoring rules; experts log in to the expert scoring system through the web page, and score the assigned evaluation tasks (expert discussion evaluation indicator system) according to the evaluation criteria, training result data, and training process data. The system will provide real-time feedback on the scoring progress and collect and integrate it;

[0171] B10. Comprehensively integrate the scoring data of each expert to obtain the evaluation results of all expert discussion evaluation indicators in the expert discussion evaluation indicator system for each evaluation object; visualize the evaluation results by calling the visualization component, report generation engine and data visualization engine through the evaluation result display and report generation subsystem. After the evaluation task is correctly executed, the user can view the evaluation results of each indicator of each evaluation object through a variety of visualization methods;

[0172] In addition, the quality of expert scoring can be evaluated based on the expert scoring results, and the authority of the scoring experts can be updated according to the evaluation results; training weakness analysis can be conducted based on the visual expert discussion evaluation results, and the weak links of the trainees in the training process can be manually analyzed, such as analyzing the impact of indicators on the evaluation results, eliminating indicators with little impact on the evaluation results, and using indicators with a large impact to evaluate the training in the next evaluation.

[0173] In the actual scoring process, statistical analysis, expert evaluation, hierarchical analysis, deduction evaluation, value center method and other methods are used, and based on the red-blue confrontation simulation results, the feasibility and advantages and disadvantages of the combat plan are evaluated in terms of the completion of the red and blue goals, combat damage or combat input.

[0174] In the process of constructing the electronic countermeasure comprehensive effectiveness index system, indicators such as radar detection, radar countermeasure, communication countermeasure, optoelectronic countermeasure, and firepower strike are selected. The selected indicators are used as a matrix, and the TOPSIS method is used to process them after matrix merging and matrix transposition, and then added to the index system; the attributes of the indicators in the electronic countermeasure comprehensive effectiveness index system are edited; the indicator attributes include information such as indicator name, indicator type, indicator unit and indicator description; the indicator types include benefit type, cost type, fixed type and deviation type; the value of each indicator has a certain influence on the electronic evaluation of the comprehensive effectiveness of electronic countermeasures; for example, radar detection is a benefit-type indicator. For the comprehensive effectiveness of electronic countermeasures, the larger its value is, the better the comprehensive effectiveness of electronic countermeasures is.

[0175] In addition, the indicators are normalized, and different types of indicators use different normalization methods. For example, the normalization formula for benefit indicators is:

[0176] ;

[0177] The normalized formula of cost-based indicators is:

[0178] ;

[0179] The normalization formula of the fixed indicator is:

[0180] ;

[0181] The normalized formula of the deviation indicator is:

[0182] ;

[0183] in, Indicates Evaluation scheme Attributes The normalized index score is Indicates Evaluation scheme Attributes The index score, represents the number of evaluation schemes, , , , The subscript sets representing benefit-type, cost-type, fixed-type and deviation-type indicators respectively, Indicates Attributes The best stable value.

[0184] After determining the specific content of the electronic countermeasure comprehensive effectiveness index system, it is necessary to set the weights of the corresponding indicators. The weight setting methods can adopt the custom method, hierarchical analysis method (AHP), chain coefficient method, entropy weight method and deviation maximization. In the process of constructing the electronic countermeasure comprehensive effectiveness index system, the weights of the indicators are calculated by using the custom method. For example, the initial values ​​of the weights of the indicators corresponding to radar detection, radar countermeasure, communication countermeasure, optoelectronic countermeasure and firepower strike are all set to 1.

[0185] After scoring by scoring experts in the expert database, the weights of each indicator under the indicator system are adjusted based on the scoring results and saved in the indicator system. For example, the initial values ​​of the indicator weights corresponding to radar detection, radar countermeasures, communication countermeasures, optoelectronic countermeasures, and firepower strike are adjusted to 0.2, 0.3, 0.1, 0.1, and 0.3 respectively.

[0186] To sum up, the present invention is based on the evaluation operation basic platform, and provides a corresponding evaluation application environment for differentiated evaluation needs of different evaluation contents, different evaluation data, different evaluation methods, and different evaluation styles, so as to support the customized evaluation needs of command training systems and simulation systems.

[0187] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-level application evaluation system, characterized in that: It includes a business layer, an evaluation basic platform layer and a support layer; the evaluation basic platform layer includes a basic subsystem, a core module and a professional database; The business layer is used to provide an application operating environment for evaluating the basic platform layer; the application operating environment includes an evaluation application operating environment based on simulation data analysis and an evaluation application operating environment based on expert discussion; The basic subsystem is used to determine the application type of the multi-level application evaluation system, and to design and manage the corresponding evaluation indicators, evaluation operators, evaluation models and evaluation tasks, and to call the core module to perform evaluation and generate a corresponding evaluation result report; the application types include command process evaluation, effect-based combat plan evaluation and command confrontation training evaluation; The core module is used to store evaluation operators and evaluation models, and provide visualization components, evaluation calculation engines, report generation engines, and data visualization engines for the basic subsystem; The professional database is used to store the basic subsystem and the data in the basic subsystem; The support layer is used to provide hardware equipment and software equipment to support the normal operation of the multi-level application evaluation system.

2. A multi-level application evaluation system based on claim 1, characterized in that: The basic subsystem includes: The data access and preprocessing subsystem is used to obtain various types of simulation data, training data, solution data, and expert evaluation data and perform preprocessing; the preprocessing includes formatting, parsing, cleaning, classification, and extraction; The evaluation task management subsystem is used to determine the application type of the multi-level application evaluation system; the application type includes evaluation purpose, evaluation background, evaluation subject, evaluation object, evaluation content, and evaluation data source; An evaluation indicator design and management subsystem, which is used to design or call corresponding indicators for evaluation of different application operating environments and different application types based on the application type, and establish a corresponding indicator system based on the called indicators; An evaluation operator design and management subsystem, used to provide corresponding algorithms for evaluation of different application operating environments and different application types based on the application type; The evaluation model design and management subsystem is used to provide corresponding evaluation models for evaluations of different application operating environments and different application types; the evaluation model is an evaluation scheme, that is, multiple algorithms selected according to the evaluations of different application operating environments and different application types; An evaluation process execution subsystem is used to call the evaluation calculation engine to run the evaluation model and obtain an evaluation result; The evaluation result display and report generation subsystem is used to call the visualization component, the report generation engine and the data visualization engine to generate and display a visualization evaluation report.

3. A multi-level application evaluation system based on claim 2, characterized in that: The core modules include: Evaluation operator library, used to store algorithms corresponding to different application operating environments and different application types; Evaluation model library, used to store evaluation plans corresponding to different application operating environments and different application types; Visual component library, used to provide visualization functions; Evaluation calculation engine, used to calculate evaluation results for different application running environments and different application types; Report generation engine, used to generate visual assessment reports; Data visualization engine for visualizing multi-tier application evaluation systems.

4. A multi-level application evaluation system based on claim 2, characterized in that: The professional database includes: Evaluation criteria library, used to formulate and store evaluation criteria; Report template library, which provides different formats for visual assessment reports; Metadatabase, used to store various simulation data, training data, solution data, and expert evaluation data; Expert database, used to store basic information of different scoring experts and set corresponding expert authority; the basic information of the scoring experts includes user name, password, profession and unit; Evaluation indicator library, used to store evaluation indicators; The evaluation indicator system library is used to store the evaluation indicator system.

5. A multi-level application evaluation method, used to implement a multi-level application evaluation system according to any one of claims 1 to 4, characterized in that: include: S1. Determine the application type through the basic subsystem; Based on the application type, providing a corresponding application operating environment through the application layer; S2. Determine hardware equipment and software equipment through the support layer; S3. Based on the application type, the data to be evaluated is obtained through the basic subsystem; the data to be evaluated is processed through the core module and the data professional library, and a corresponding visual evaluation result report is generated and displayed.

6. A multi-level application evaluation method based on claim 5, characterized in that: The basic subsystem includes a data access and preprocessing subsystem, an evaluation task management subsystem, an evaluation indicator design and management subsystem, an evaluation operator design and management subsystem, an evaluation model design and management subsystem, an evaluation process execution subsystem, and an evaluation result display and report generation subsystem; The core modules include an evaluation operator library, an evaluation model library, a visualization component library, an evaluation calculation engine, a report generation engine, and a data visualization engine; The professional database includes an evaluation standard library, a report template library, a metadata library, an evaluation index library and an evaluation index system library.

7. A multi-level application evaluation method based on claim 6, characterized in that: Based on the application type, obtaining the data to be evaluated through the basic subsystem; The data to be evaluated is processed through the core module and the data professional library to generate and display a corresponding visual evaluation result report, including: S3-1, obtaining all corresponding evaluation data through the data access and preprocessing subsystem, preprocessing them, and transmitting them to the metadata database for storage; S3-2, selecting experts through the expert database; S3-3, selecting the preprocessed evaluation data through the evaluation index design and management subsystem and the selected experts to obtain corresponding evaluation index data, and transmitting the data to the evaluation index library for storage; S3-4, based on the evaluation indicator data, formulate corresponding evaluation standards through the evaluation indicator design and management subsystem, and transmit to the evaluation standard library for storage; S3-5, based on the evaluation criteria, construct a corresponding evaluation indicator system through the evaluation indicator design and management subsystem, and transmit it to the evaluation indicator system library for storage; S3-6, determining the application type through the assessment task management subsystem; S3-7, based on the application type, calling the corresponding evaluation indicator system in the evaluation indicator system through the evaluation indicator design and management subsystem; S3-8, based on the application type and the called evaluation indicator system, calling the algorithm in the evaluation operator library through the evaluation operator design and management subsystem; S3-9, determining the order or process of the called algorithms through the evaluation model design and management subsystem, obtaining the corresponding evaluation model, and transmitting it to the evaluation model library for storage; the evaluation model is an evaluation plan; S3-10, calling the evaluation calculation engine through the evaluation process execution subsystem to execute the evaluation plan and obtain the corresponding evaluation result; S3-11. Based on the application type, select a corresponding report template from the report templates; S3-12. Based on the selected report template, the evaluation result display and report generation subsystem calls the visualization component, the report generation engine and the data visualization engine to visualize the evaluation result, generate a visual evaluation report and display it.

8. A multi-level application evaluation method based on claim 7, characterized in that: In S3-7, the application type is matched with the evaluation indicator system in the evaluation indicator system. If the matching result is: there is a corresponding evaluation indicator system, then enter S3-8; if the matching result is: there is no corresponding evaluation indicator system, then enter the following steps: Based on the application type, corresponding data to be evaluated is obtained, corresponding evaluation indicators, evaluation standards and evaluation indicator systems are formulated through the evaluation indicator design and management subsystem, and transmitted to the evaluation indicator library, the evaluation standard library and the evaluation indicator system library respectively; Based on the application type and the called evaluation index system, the algorithm in the evaluation operator library is called through the evaluation operator design and management subsystem; the order or process of the called algorithm is determined through the evaluation model design and management subsystem, the corresponding evaluation model is obtained, and it is transmitted to the evaluation model library for storage; enter S3-10.

Images