Intelligent information system performance evaluation method, system, computing device and storage medium

Abstract

The present application discloses an intelligent information system performance evaluation method, system, computing device and storage medium, and relates to the technical field of computers. The method comprises: obtaining evaluation requirements, and gradually decomposing the evaluation requirements to obtain an indicator hierarchy structure; wherein the indicator hierarchy structure comprises a plurality of hierarchical indicators and an inclusion relationship between the hierarchical indicators; screening the hierarchical indicators, taking a plurality of screened hierarchical indicators as first target indicators, and creating an indicator tree according to the first target indicator; the indicator tree comprises an inclusion relationship between a plurality of the first target indicators and the first target indicator; assigning a corresponding indicator weight to each node of the indicator tree; generating a first evaluation questionnaire according to the terminal node of the indicator tree, and performing evaluation according to the first evaluation questionnaire to obtain an evaluation value corresponding to the terminal node of the indicator tree; the present application has the effect of improving the accuracy of the evaluation result.

Description

Technical Field

[0001] The present application relates to the technical field of computers, and in particular to an intelligent information system performance evaluation method, system, computing device and storage medium. Background Art

[0002] System effectiveness refers to the ability of a system to achieve specified usage objectives under specified conditions. The specified conditions refer to factors such as environmental conditions, time, personnel, and usage methods. The specified usage objectives refer to the goals to be achieved. Capability refers to the quantitative or qualitative degree of achieving the goals. System effectiveness is a comprehensive reflection of the system's availability, reliability, and inherent capabilities.

[0003] At present, when evaluating the performance of a system, usually only the factors of the system itself are taken into consideration, that is, the relevant data for the evaluation comes from the system itself, and the source of the relevant data for the evaluation is relatively single, which makes it difficult for the relevant data for the evaluation to fully reflect the functions of the system itself, making it difficult for the final evaluation results to fully reflect the performance of the system, and the evaluation results are not accurate enough. Summary of the invention

[0004] In order to improve the accuracy of evaluation results, the present application provides an intelligent information system performance evaluation method, system, computing device and storage medium.

[0005] In a first aspect, the present application provides an intelligent information system performance evaluation method, which adopts the following technical solution.

[0006] Intelligent information system effectiveness evaluation methods, including,

[0007] Acquire evaluation requirements, and gradually decompose the evaluation requirements to obtain an indicator hierarchical structure; wherein the indicator hierarchical structure includes a plurality of hierarchical indicators and an inclusion relationship between the hierarchical indicators;

[0008] The hierarchical indicators are screened, a number of the screened hierarchical indicators are used as first target indicators, and an indicator tree is created according to the first target indicators; the indicator tree includes a plurality of inclusion relationships between the first target indicators and the first target indicators;

[0009] Assigning a corresponding indicator weight to each node of the indicator tree;

[0010] Generate a first evaluation questionnaire according to the terminal node of the indicator tree, and perform evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree;

[0011] Calculate a first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value;

[0012] Calling a scale generation interface to generate an evaluation scale, and performing evaluation according to the evaluation scale to obtain a second target indicator and a second evaluation value corresponding to the second target indicator;

[0013] Acquire multimodal evaluation data, and determine a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal evaluation data;

[0014] Based on the evaluation requirements, an evaluation tree is constructed according to the first target indicator, the second target indicator and the third target indicator corresponding to the root node of the indicator tree;

[0015] Normalizing the first evaluation value, the second evaluation value, and the third evaluation value to obtain corresponding first evaluation value, second evaluation value, and third evaluation value;

[0016] Assigning a corresponding evaluation weight to each node of the evaluation tree;

[0017] Calculate the validity value of each node of the evaluation tree according to the first evaluation value, the second evaluation value, the third evaluation value and the evaluation weight;

[0018] An effectiveness evaluation report is generated according to the evaluation tree and the validity value corresponding to each node of the evaluation tree.

[0019] By adopting the above technical solution, when performance evaluation is required, the evaluation requirements are first obtained, and then the evaluation requirements are gradually decomposed to obtain the indicator hierarchical structure, and then the hierarchical indicators are screened, and several screened hierarchical indicators are used as the first target indicators, and an indicator tree is created according to the first target indicator, and then each node of the indicator tree is assigned a corresponding indicator weight, and then a first evaluation questionnaire is generated according to the terminal node of the indicator tree, and the evaluation is performed according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree, and then the first evaluation value corresponding to each node of the indicator tree is calculated according to the indicator tree, the indicator weight and the evaluation value, and then the scale generation interface is called to generate an evaluation scale, and the evaluation is performed according to the evaluation scale to obtain the second target indicator and the second The second evaluation value corresponding to the target indicator, then obtain the multimodal evaluation data, and determine the third target indicator and the third evaluation value corresponding to the third target indicator according to the multimodal evaluation data, then based on the evaluation requirements, and according to the first target indicator, the second target indicator and the third target indicator corresponding to the root node of the indicator tree, construct an evaluation tree, then normalize the first evaluation value, the second evaluation value and the third evaluation value to obtain the corresponding first evaluation value, the second evaluation value and the third evaluation value, then assign a corresponding evaluation weight to each node of the evaluation tree, then calculate the validity value of each node of the evaluation tree according to the first evaluation value, the second evaluation value, the third evaluation value and the evaluation weight, and generate an effectiveness evaluation report according to the evaluation tree and the validity value corresponding to each node of the evaluation tree. In the above method, the generated evaluation tree integrates the evaluation results of the indicator tree, the evaluation scale and the multimodal evaluation data, that is, integrates the first target indicator, the second target indicator and the third target indicator, and links the generation of the first target indicator, the second target indicator and the third target indicator with the participation of the evaluator, reduces the limitations of the system itself, makes the generation of the effectiveness evaluation report more accurate, thereby improving the accuracy of the evaluation results.

[0020] Optionally, calculating the first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value includes:

[0021] For any terminal node X1 of the index tree,

[0022] S1=W*X

[0023] Wherein, S1 is the first evaluation value corresponding to the terminal node X1 of the indicator tree, W is the indicator weight corresponding to the terminal node X1 of the indicator tree, and X is the evaluation value corresponding to the terminal node X1 of the indicator tree;

[0024] For any non-terminal node X2 of the index tree,

[0025]

[0026] Wherein, S2 is the first evaluation value corresponding to the non-terminal node X2 of the index tree, n is the number of child nodes contained in the non-terminal node X2 of the index tree, and W i is the index weight corresponding to the child node contained in the non-terminal node X2 of the index tree, X i It is the first evaluation value corresponding to the child node contained in the non-terminal node X2 of the indicator tree.

[0027] Optionally, the normalizing the first evaluation value, the second evaluation value, and the third evaluation value includes:

[0028]

[0029] Among them, Z x is one of the first evaluation value, the second evaluation value and the third evaluation value, Z′ x For Z x The value after normalization processing, m is the total number of the first evaluation value, the second evaluation value and the third evaluation value, and y is the preset range value.

[0030] Intelligent information system effectiveness evaluation method, including:

[0031] Acquire evaluation requirements, and gradually decompose the evaluation requirements to obtain an indicator hierarchical structure; wherein the indicator hierarchical structure includes a plurality of hierarchical indicators and an inclusion relationship between the hierarchical indicators;

[0032] The hierarchical indicators are screened, a number of the screened hierarchical indicators are used as first target indicators, and an indicator tree is created according to the first target indicators; the indicator tree includes a plurality of inclusion relationships between the first target indicators and the first target indicators;

[0033] Assigning a corresponding indicator weight to each node of the indicator tree;

[0034] Generate a first evaluation questionnaire according to the terminal node of the indicator tree, and perform evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree;

[0035] Calculate a first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value;

[0036] Calling a scale generation interface to generate an evaluation scale, and performing evaluation according to the evaluation scale to obtain a second target indicator and a second evaluation value corresponding to the second target indicator;

[0037] Based on the evaluation requirements, and according to the first target indicator and the second target indicator corresponding to the root node of the indicator tree, an evaluation tree is constructed;

[0038] Normalizing the first evaluation value and the second evaluation value to obtain corresponding first evaluation value and second evaluation value; assigning corresponding evaluation weights to each node of the evaluation tree;

[0039] Calculate the validity value of each node of the evaluation tree according to the first evaluation value, the second evaluation value and the evaluation weight;

[0040] An effectiveness evaluation report is generated according to the evaluation tree and the validity value corresponding to each node of the evaluation tree.

[0041] By adopting the above technical solution, the generated evaluation tree integrates the evaluation results of the indicator tree and the evaluation scale, that is, it integrates the first target indicator and the second target indicator, and links the generation of the first target indicator and the second target indicator with the participation of the evaluators, thereby reducing the limitations of the system itself and making the generated performance evaluation report more accurate, thereby improving the accuracy of the evaluation results.

[0042] Intelligent information system effectiveness evaluation method, including:

[0043] Acquire evaluation requirements, and gradually decompose the evaluation requirements to obtain an indicator hierarchical structure; wherein the indicator hierarchical structure includes a plurality of hierarchical indicators and an inclusion relationship between the hierarchical indicators;

[0044] The hierarchical indicators are screened, a number of the screened hierarchical indicators are used as first target indicators, and an indicator tree is created according to the first target indicators; the indicator tree includes a plurality of inclusion relationships between the first target indicators and the first target indicators;

[0045] Assigning a corresponding indicator weight to each node of the indicator tree;

[0046] Generate a first evaluation questionnaire according to the terminal node of the indicator tree, and perform evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree;

[0047] Calculate a first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value;

[0048] Acquire multimodal evaluation data, and determine a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal evaluation data;

[0049] Based on the evaluation requirements, and according to the first target indicator and the third target indicator corresponding to the root node of the indicator tree, an evaluation tree is constructed;

[0050] Normalizing the first evaluation value and the third evaluation value to obtain corresponding first evaluation value and third evaluation value; assigning corresponding evaluation weights to each node of the evaluation tree;

[0051] Calculate the validity value of each node of the evaluation tree according to the first evaluation value, the third evaluation value and the evaluation weight;

[0052] An effectiveness evaluation report is generated according to the evaluation tree and the validity value corresponding to each node of the evaluation tree.

[0053] By adopting the above technical solution, the generated evaluation tree integrates the indicator tree and multimodal evaluation data, that is, it integrates the first target indicator and the third target indicator, and links the generation of the first target indicator and the third target indicator with the participation of evaluators, thereby reducing the limitations of the system itself and making the generated performance evaluation report more accurate, thereby improving the accuracy of the evaluation results.

[0054] Intelligent information system effectiveness evaluation method, including:

[0055] Calling a scale generation interface to generate an evaluation scale, and performing evaluation according to the evaluation scale to obtain a second target indicator and a second evaluation value corresponding to the second target indicator;

[0056] Acquire multimodal evaluation data, and determine a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal evaluation data;

[0057] Based on the evaluation requirements, and according to the second target indicator and the third target indicator, construct an evaluation tree;

[0058] Normalizing the second evaluation value and the third evaluation value to obtain corresponding second evaluation value and third evaluation value; assigning corresponding evaluation weights to each node of the evaluation tree;

[0059] Calculate the validity value of each node of the evaluation tree according to the second evaluation value, the third evaluation value and the evaluation weight;

[0060] An effectiveness evaluation report is generated according to the evaluation tree and the validity value corresponding to each node of the evaluation tree.

[0061] By adopting the above technical solution, the generated evaluation tree integrates the evaluation results of the evaluation scale and the multimodal evaluation data, that is, it integrates the second target indicator and the third target indicator, and links the generation of the second target indicator and the third target indicator with the participation of the evaluators, thereby reducing the limitations of the system itself and making the generated performance evaluation report more accurate, thereby improving the accuracy of the evaluation results.

[0062] In the second aspect, the present application provides an intelligent information system performance evaluation system, which adopts the following technical solution.

[0063] Intelligent information system effectiveness evaluation system, including:

[0064] An evaluation requirement decomposition module is used to obtain evaluation requirements and gradually decompose the evaluation requirements to obtain an indicator hierarchical structure; wherein the indicator hierarchical structure includes a plurality of hierarchical indicators and an inclusion relationship between the hierarchical indicators;

[0065] An indicator tree generation module, used to screen the hierarchical indicators, use the screened hierarchical indicators as first target indicators, and create an indicator tree according to the first target indicators; wherein the indicator tree includes an inclusion relationship between the first target indicators and the first target indicators;

[0066] An indicator weight assignment module, used to assign a corresponding indicator weight to each node of the indicator tree;

[0067] A first evaluation module, used to generate a first evaluation questionnaire according to the terminal node of the indicator tree, and perform evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree;

[0068] A first evaluation value generating module, used for calculating a first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value;

[0069] A second evaluation value generation module is used to call a scale generation interface to generate an evaluation scale, and to perform evaluation according to the evaluation scale to obtain a second target indicator and a second evaluation value corresponding to the second target indicator;

[0070] A third evaluation value generating module, used to obtain multimodal data, and determine a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal evaluation data;

[0071] An evaluation tree generation module, used to construct an evaluation tree based on the evaluation requirements and according to the first target indicator, the second target indicator and the third target indicator corresponding to the root node of the indicator tree;

[0072] A normalization processing module, used for performing normalization processing on the first evaluation value, the second evaluation value and the third evaluation value to obtain corresponding first evaluation value, second evaluation value and third evaluation value;

[0073] An evaluation weight assigning module, used to assign a corresponding evaluation weight to each node of the evaluation tree;

[0074] A validity value generating module, used for calculating the validity value of each node of the evaluation tree according to the first evaluation value, the second evaluation value, the third evaluation value and the evaluation weight;

[0075] The effectiveness evaluation report generating module is used to generate an effectiveness evaluation report according to the evaluation tree and the validity value corresponding to each node of the evaluation tree.

[0076] In a third aspect, the present application provides a computing device that adopts the following technical solution.

[0077] A computing device comprises a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements any one of the methods described in the first aspect when executing the computer program.

[0078] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution.

[0079] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the methods described in the first aspect.

[0080] To summarize, the present application includes at least the following beneficial technical effects: in the present application, the generated evaluation tree integrates the indicator tree, the evaluation results of the evaluation scale and the multimodal evaluation data, that is, it integrates the first target indicator, the second target indicator and the third target indicator, and links the generation of the first target indicator, the second target indicator and the third target indicator with the participation of the evaluators, thereby reducing the limitations of the system itself and making the generated performance evaluation report more accurate, thereby improving the accuracy of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a flow chart of one implementation method of Example 1 of the present application.

[0082] Figure 2 This is a diagram of the visual expression of Example 1 of the present application.

[0083] Figure 3 It is a flow chart of another implementation of Example 1 of the present application.

[0084] Figure 4It is a flow chart of another implementation method of Example 1 of the present application.

[0085] Figure 5 It is a flowchart of other implementation methods of Example 1 of the present application.

[0086] Figure 6 It is a structural block diagram of Example 2 of the present application.

[0087] Figure 7 It is a structural block diagram of Example 3 of the present application. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0089] Embodiment 1:

[0090] The embodiment of the present application discloses a method for evaluating the effectiveness of an intelligent information system.

[0091] refer to Figure 1 ,The ,intelligent information system effectiveness evaluation method includes the following ,steps:

[0092] Step S1, obtaining the evaluation requirements, and gradually decomposing the evaluation requirements to obtain an indicator hierarchical structure.

[0093] Among them, the indicator hierarchical structure includes the inclusion relationship between several hierarchical indicators and hierarchical indicators.

[0094] It should be noted that in the process of decomposing the evaluation requirements, the evaluators can determine the direction of decomposition through the functions displayed on the interface of the evaluated system or by asking the system designers, etc., to understand the indicator requirements of the evaluated system, and define and name the corresponding indicators according to the indicator requirements, and obtain the inclusion relationship between several hierarchical indicators, that is, the indicator hierarchy structure.

[0095] Step S2, screening the hierarchical indicators, taking a number of screened hierarchical indicators as first target indicators, and creating an indicator tree according to the first target indicators.

[0096] The indicator tree includes a plurality of inclusion relationships between first target indicators and first target indicators.

[0097] It should be noted that there are usually two ways of screening. One is subjective screening, that is, the evaluator screens out the hierarchical indicators that are more relevant to the evaluation needs based on his own experience, expert advice, etc. The other is algorithmic screening, that is, the evaluator inputs the hierarchical indicators into the system, and the system screens them through the built-in screening algorithm. Common screening algorithms include the Delphi algorithm.

[0098] Step S3, assigning a corresponding indicator weight to each node of the indicator tree.

[0099] It should be noted that there are usually two ways to assign indicator weights. One is subjective weighting, that is, the evaluator assigns corresponding indicator weights to the nodes of the indicator tree according to the importance of the first target indicator corresponding to the nodes of the indicator tree. The other is algorithmic weighting. Common weighting algorithms include G1 method and new G1 method.

[0100] It should be further explained that in the indicator tree, the indicator weights corresponding to the child nodes belonging to the same node X should satisfy:

[0101]

[0102] Where i is the number of child nodes of node X, W i is the indicator weight corresponding to the child node of node X.

[0103] Step S4, generating a first evaluation questionnaire according to the terminal node of the indicator tree, and performing evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree.

[0104] It should be noted that the first evaluation questionnaire includes information on the first target indicator corresponding to the terminal node of the indicator tree. During the evaluation process, the first target indicator will be scored according to its importance in the first evaluation questionnaire, and the scored value is the evaluation value.

[0105] Step S5, calculating the first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value.

[0106] Specifically, according to the indicator tree, the indicator weight and the evaluation value, the first evaluation value corresponding to each node of the indicator tree is calculated, including:

[0107] For any terminal node X1 of the index tree,

[0108] S1=W*X

[0109] Among them, S1 is the first evaluation value corresponding to the terminal node X1 of the indicator tree, W is the indicator weight corresponding to the terminal node X1 of the indicator tree, and X is the evaluation value corresponding to the terminal node X1 of the indicator tree;

[0110] For any non-terminal node X2 of the index tree,

[0111]

[0112] Where S2 is the first evaluation value corresponding to the non-terminal node X2 of the index tree, n is the number of child nodes contained in the non-terminal node X2 of the index tree, and W i is the index weight corresponding to the child nodes contained in the non-terminal node X2 of the index tree, X i It is the first evaluation value corresponding to the child node contained in the non-terminal node X2 of the index tree.

[0113] Step S6, calling the scale generation interface to generate an evaluation scale, and performing evaluation according to the evaluation scale to obtain a second target indicator and a second evaluation value corresponding to the second target indicator.

[0114] It should be noted that the evaluation system has built-in scales such as the NASA_TLX workload scale, the SART situational awareness scale, and the SUS availability scale. After calling the scale generation interface, the evaluation system generates an evaluation scale based on the above-mentioned built-in scales; after the evaluation, the evaluation system will process the relevant data of the evaluation to obtain the second target indicator and the corresponding second evaluation value.

[0115] Step S7, obtaining multimodal evaluation data, and determining a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal evaluation data.

[0116] It should be noted that, in the present embodiment, multimodal evaluation data is generated through models such as the mathematical prototype, VR prototype, physical prototype or real prototype of the system being evaluated; the evaluator can customize the third target indicator and the indicator data corresponding to the third target indicator of the evaluation system. After the setting is completed, the evaluation system calculates the third evaluation value corresponding to the third target indicator through the indicator data corresponding to the third target indicator.

[0117] It should be further explained that the third target indicators include comfort, visibility, accessibility, situational awareness, etc. The types of indicator data corresponding to the third target indicators include physiological, EEG, eye movement, interactive behavior, near infrared, environment, etc.

[0118] Step S8, based on the evaluation requirements, and according to the first target indicator, the second target indicator and the third target indicator corresponding to the root node of the indicator tree, construct an evaluation tree.

[0119] It should be noted that in the process of constructing the evaluation tree, the first target indicator, the second target indicator and the third target indicator corresponding to the root node of the indicator tree serve as the terminal nodes of the evaluation tree. The evaluator can set the non-terminal nodes of the evaluation tree based on the evaluation requirements. For example, the combat effectiveness evaluation report of the human-information-physical system of UAV combat control has three sub-nodes: man-machine collaborative combat capability coefficient, situational awareness coefficient, and mission execution coefficient. The man-machine collaborative combat capability coefficient has four sub-nodes: system automation degree, cognitive load degree, training cost, and operation task efficiency. The situational awareness coefficient has two sub-nodes: system SA and operator SA. The mission execution system has three sub-nodes: system anti-interference capability, system response time, and reconnaissance accuracy. At the same time, the nine sub-nodes: system automation degree, cognitive load degree, training cost, operation task efficiency, system SA, operator SA, system anti-interference capability, system response time, and reconnaissance accuracy are all terminal nodes.

[0120] Step S9, normalizing the first evaluation value, the second evaluation value, and the third evaluation value to obtain corresponding first evaluation value, second evaluation value, and third evaluation value.

[0121] The first evaluation value, the second evaluation value and the third evaluation value are normalized, including:

[0122]

[0123] Among them, Z x is one of the first evaluation value, the second evaluation value and the third evaluation value, Z′ x For Z x The value after normalization processing, m is the total number of the first evaluation value, the second evaluation value and the third evaluation value, and y is the preset range value.

[0124] It should be noted that y is the preset range value, Z′ x The value of Z is between 0 and y. For example, when y is 100, ′ x The value of is between 0 and 100.

[0125] It should be further explained that each terminal node of the evaluation tree corresponds to one or more of the first evaluation value, the second evaluation value and the third evaluation value.

[0126] Step S10, assigning a corresponding evaluation weight to each node of the evaluation tree.

[0127] It should be noted that there are usually two ways to assign evaluation weights, one is subjective weighting and the other is algorithmic weighting. The specific weighting requirements are the same as those in step S3.

[0128] Step S11, calculating the validity value of each node of the evaluation tree according to the first evaluation value, the second evaluation value, the third evaluation value and the evaluation weight.

[0129] Among them, for any terminal node Y1 of the evaluation tree,

[0130] E=W*Y

[0131] Wherein, for the convenience of expression, the first evaluation value, the second evaluation value, and the third evaluation value are collectively referred to as evaluation values; E is the validity value corresponding to the terminal node Y1 of the indicator tree, W is the evaluation weight corresponding to the terminal node Y1 of the indicator tree, and Y is the evaluation value corresponding to the terminal node Y1 of the indicator tree;

[0132] For any non-terminal node Y2 of the evaluation tree,

[0133]

[0134] Where E is the validity value corresponding to the non-terminal node Y2 of the evaluation tree, n is the number of child nodes contained in the non-terminal node Y2 of the evaluation tree, and W i is the evaluation weight corresponding to the child nodes contained in the non-terminal node Y2 of the evaluation tree, E i It is the validity value corresponding to the child nodes contained in the non-terminal node Y2 of the evaluation tree.

[0135] Step S12: Generate an effectiveness evaluation report according to the evaluation tree and the validity value corresponding to each node of the evaluation tree.

[0136] It can be understood that, for the sake of convenience, the first target indicator, the second target indicator and the third target indicator corresponding to the node of the evaluation tree are collectively referred to as the target indicator. The performance evaluation report includes the target indicator and the validity value corresponding to the target indicator. The performance evaluation report includes a table, a bar chart and other visual expressions. Figure 2 , the assessors passed Figure 2 Ability to understand the effectiveness of the system being evaluated.

[0137] In the above implementation, when performance evaluation is required, the evaluation requirements are first obtained, and then the evaluation requirements are gradually decomposed to obtain an indicator hierarchical structure, and then the hierarchical indicators are screened, and several screened hierarchical indicators are used as the first target indicators, and an indicator tree is created based on the first target indicator, and then each node of the indicator tree is assigned a corresponding indicator weight, and then a first evaluation questionnaire is generated based on the terminal node of the indicator tree, and an evaluation is performed based on the first evaluation questionnaire to obtain an evaluation value corresponding to the terminal node of the indicator tree, and then the first evaluation value corresponding to each node of the indicator tree is calculated based on the indicator tree, the indicator weight and the evaluation value, and then the scale generation interface is called to generate an evaluation scale, and the evaluation is performed based on the evaluation scale to obtain a second target indicator and a second target indicator. The second evaluation value corresponding to the target indicator is obtained, and then the multimodal evaluation data is obtained, and the third target indicator and the third evaluation value corresponding to the third target indicator are determined according to the multimodal evaluation data, and then the evaluation tree is constructed based on the evaluation requirements and the first target indicator, the second target indicator and the third target indicator corresponding to the root node of the indicator tree, and then the first evaluation value, the second evaluation value and the third evaluation value are normalized to obtain the corresponding first evaluation value, the second evaluation value and the third evaluation value, and then each node of the evaluation tree is assigned a corresponding evaluation weight, and then the validity value of each node of the evaluation tree is calculated according to the first evaluation value, the second evaluation value, the third evaluation value and the evaluation weight, and the effectiveness evaluation report is generated according to the evaluation tree and the validity value corresponding to each node of the evaluation tree. In the above method, the generated evaluation tree integrates the evaluation results of the indicator tree, the evaluation scale and the multimodal evaluation data, that is, it integrates the first target indicator, the second target indicator and the third target indicator, and the generation of the first target indicator, the second target indicator and the third target indicator is linked to the participation of the evaluator, which reduces the limitations of the system itself, makes the generation of the effectiveness evaluation report more accurate, and thus improves the accuracy of the evaluation results.

[0138] refer to Figure 3 As another implementation of the performance evaluation method, the performance evaluation method includes the following steps:

[0139] Step S20, obtaining the evaluation requirements, and gradually decomposing the evaluation requirements to obtain an indicator hierarchical structure.

[0140] Among them, the indicator hierarchy structure includes the inclusion relationship between several hierarchical indicators and hierarchical indicators.

[0141] Step S21, screen the hierarchical indicators, use several screened hierarchical indicators as the first target indicators, and create an indicator tree based on the first target indicators.

[0142] The indicator tree includes a plurality of inclusion relationships between first target indicators and first target indicators.

[0143] Step S22, assigning a corresponding indicator weight to each node of the indicator tree.

[0144] Step S23, generating a first evaluation questionnaire according to the terminal node of the indicator tree, and performing evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree.

[0145] Step S24, calculating the first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value.

[0146] Step S25, calling the scale generation interface to generate an evaluation scale, and performing evaluation according to the evaluation scale to obtain a second target indicator and a second evaluation value corresponding to the second target indicator.

[0147] Step S26: construct an evaluation tree based on the evaluation requirements and according to the first target indicator and the second target indicator corresponding to the root node of the indicator tree.

[0148] Step S27, normalizing the first evaluation value and the second evaluation value to obtain corresponding first evaluation value and second evaluation value.

[0149] Step S28, assigning a corresponding evaluation weight to each node of the evaluation tree.

[0150] Step S29, calculating the validity value of each node of the evaluation tree according to the first evaluation value, the second evaluation value and the evaluation weight.

[0151] Step S30, generating an effectiveness evaluation report according to the evaluation tree and the validity value corresponding to each node of the evaluation tree.

[0152] In the above implementation, the generated evaluation tree integrates the evaluation results of the indicator tree and the evaluation scale, that is, it integrates the first target indicator and the second target indicator, and links the generation of the first target indicator and the second target indicator with the participation of the evaluators, thereby reducing the limitations of the system itself and making the generated performance evaluation report more accurate, thereby improving the accuracy of the evaluation results.

[0153] refer to Figure 4 As another implementation of the performance evaluation method, the performance evaluation method includes the following steps:

[0154] Step S40, obtaining the evaluation requirements, and gradually decomposing the evaluation requirements to obtain an indicator hierarchical structure.

[0155] Among them, the indicator hierarchy structure includes the inclusion relationship between several hierarchical indicators and hierarchical indicators.

[0156] Step S41, screening the hierarchical indicators, taking a number of screened hierarchical indicators as first target indicators, and creating an indicator tree according to the first target indicators.

[0157] The indicator tree includes a plurality of inclusion relationships between first target indicators and first target indicators.

[0158] Step S42: assign corresponding indicator weights to each node of the indicator tree.

[0159] Step S43, generating a first evaluation questionnaire according to the terminal node of the indicator tree, and performing evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree.

[0160] Step S44, calculating the first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value.

[0161] Step S45, obtaining multimodal evaluation data, and determining a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal evaluation data.

[0162] Step S46: construct an evaluation tree based on the evaluation requirements and according to the first target indicator and the third target indicator corresponding to the root node of the indicator tree.

[0163] Step S47, normalizing the first evaluation value and the third evaluation value to obtain corresponding first evaluation value and third evaluation value.

[0164] Step S48: assign a corresponding evaluation weight to each node of the evaluation tree.

[0165] Step S49, calculating the validity value of each node of the evaluation tree according to the first evaluation value, the third evaluation value and the evaluation weight.

[0166] Step S50: Generate an effectiveness evaluation report according to the evaluation tree and the validity value corresponding to each node of the evaluation tree.

[0167] In the above implementation, the generated evaluation tree integrates the indicator tree and multimodal evaluation data, that is, it integrates the first target indicator and the third target indicator, and links the generation of the first target indicator and the third target indicator with the participation of evaluators, thereby reducing the limitations of the system itself and making the generated performance evaluation report more accurate, thereby improving the accuracy of the evaluation results.

[0168] refer to Figure 5 As another implementation of the performance evaluation method, the performance evaluation method includes the following steps:

[0169] Step S60, calling a scale generation interface to generate an evaluation scale, and performing evaluation according to the evaluation scale to obtain a second target indicator and a second evaluation value corresponding to the second target indicator.

[0170] Step S61, obtaining multimodal evaluation data, and determining a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal evaluation data.

[0171] Step S62: construct an evaluation tree based on the evaluation requirements and in accordance with the second target indicator and the third target indicator.

[0172] Step S63, normalizing the second evaluation value and the third evaluation value to obtain corresponding second evaluation value and third evaluation value.

[0173] Step S64: assign a corresponding evaluation weight to each node of the evaluation tree.

[0174] Step S65, calculating the validity value of each node of the evaluation tree according to the second evaluation value, the third evaluation value and the evaluation weight.

[0175] Step S66, generating an effectiveness evaluation report according to the evaluation tree and the validity value corresponding to each node of the evaluation tree.

[0176] In the above implementation, the generated evaluation tree integrates the evaluation results of the evaluation scale and the multimodal evaluation data, that is, it integrates the second target indicator and the third target indicator, and links the generation of the second target indicator and the third target indicator with the participation of the evaluators, thereby reducing the limitations of the system itself and making the generated performance evaluation report more accurate, thereby improving the accuracy of the evaluation results.

[0177] Embodiment 2:

[0178] The embodiment of the present application discloses an intelligent information system performance evaluation system.

[0179] refer to Figure 6 , intelligent information system effectiveness evaluation system, including:

[0180] The evaluation requirement decomposition module 1 is used to obtain the evaluation requirements and gradually decompose the evaluation requirements to obtain an indicator hierarchical structure; wherein the indicator hierarchical structure includes a plurality of inclusion relationships between hierarchical indicators;

[0181] The indicator tree generation module 2 is used to screen the hierarchical indicators, use the screened hierarchical indicators as the first target indicators, and create an indicator tree according to the first target indicators; wherein the indicator tree includes the inclusion relationship between the first target indicators and the first target indicators;

[0182] The indicator weight assignment module 3 is used to assign a corresponding indicator weight to each node of the indicator tree;

[0183] The first evaluation module 4 is used to generate a first evaluation questionnaire according to the terminal node of the indicator tree, and to perform evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree;

[0184] A first evaluation value generating module 5 is used to calculate a first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value;

[0185] The second evaluation value generating module 6 is used to call the scale generating interface to generate an evaluation scale including a plurality of second target indicators, and to perform evaluation according to the evaluation scale to obtain a second evaluation value corresponding to the second target indicator;

[0186] A third evaluation value generating module 7, used for acquiring multimodal data, and determining a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal data;

[0187] An evaluation tree generation module 8 is used to construct an evaluation tree based on the evaluation requirements and according to the first target indicator, the second target indicator and the third target indicator corresponding to the root node of the indicator tree;

[0188] A normalization processing module 9 is used to perform normalization processing on the first evaluation value, the second evaluation value and the third evaluation value to obtain the corresponding first evaluation value, the second evaluation value and the third evaluation value;

[0189] An evaluation weight assigning module 10 is used to assign a corresponding evaluation weight to each node of the evaluation tree;

[0190] A validity value generating module 11 is used to calculate the validity value of each node of the evaluation tree according to the first evaluation value, the second evaluation value, the third evaluation value and the evaluation weight;

[0191] The effectiveness evaluation report generating module 12 is used to generate an effectiveness evaluation report according to the evaluation tree and the effectiveness value corresponding to each node of the evaluation tree.

[0192] The intelligent information system performance evaluation system of the present invention can implement any of the above-mentioned intelligent information system performance evaluation methods, and the specific working process of the intelligent information system performance evaluation system of the present invention can refer to the corresponding process in the above-mentioned intelligent information system performance evaluation method.

[0193] Embodiment 3:

[0194] An embodiment of the present application discloses a computing device.

[0195] refer to Figure 7 , a computing device includes a memory and a processor, the memory stores a computer program that can be run on the processor, and the processor implements any of the above methods when executing the computer program.

[0196] Embodiment 4:

[0197] An embodiment of the present application discloses a computer-readable storage medium.

[0198] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any of the above methods. The computer-readable storage medium of this embodiment includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0199] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. An intelligent information system performance evaluation method, characterized in that: include, Acquire evaluation requirements, and gradually decompose the evaluation requirements to obtain an indicator hierarchical structure; wherein the indicator hierarchical structure includes a plurality of hierarchical indicators and an inclusion relationship between the hierarchical indicators; The hierarchical indicators are screened, a number of the screened hierarchical indicators are used as first target indicators, and an indicator tree is created according to the first target indicators; the indicator tree includes a plurality of inclusion relationships between the first target indicators and the first target indicators; Assigning a corresponding indicator weight to each node of the indicator tree; Generate a first evaluation questionnaire according to the terminal node of the indicator tree, and perform evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree; Calculate a first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value; Calling a scale generation interface to generate an evaluation scale, and performing evaluation according to the evaluation scale to obtain a second target indicator and a second evaluation value corresponding to the second target indicator; Acquire multimodal evaluation data, and determine a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal evaluation data, wherein the multimodal evaluation data is generated by a mathematical prototype, a VR prototype, a physical prototype or a real prototype of the system being evaluated; Based on the evaluation requirements, an evaluation tree is constructed according to the first target indicator, the second target indicator and the third target indicator corresponding to the root node of the indicator tree; Normalizing the first evaluation value, the second evaluation value, and the third evaluation value to obtain corresponding first evaluation value, second evaluation value, and third evaluation value; Assigning a corresponding evaluation weight to each node of the evaluation tree; Calculate the validity value of each node of the evaluation tree according to the first evaluation value, the second evaluation value, the third evaluation value and the evaluation weight; Generate an effectiveness evaluation report according to the evaluation tree and the validity value corresponding to each node of the evaluation tree, The step of calculating the first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value includes: For any terminal node X1 of the index tree, S1=W*X Wherein, S1 is the first evaluation value corresponding to the terminal node X1 of the indicator tree, W is the indicator weight corresponding to the terminal node X1 of the indicator tree, and X is the evaluation value corresponding to the terminal node X1 of the indicator tree; For any non-terminal node X2 of the index tree, Wherein, S2 is the first evaluation value corresponding to the non-terminal node X2 of the index tree, n is the number of child nodes contained in the non-terminal node X2 of the index tree, and W i is the index weight corresponding to the child node contained in the non-terminal node X2 of the index tree, X i is the first evaluation value corresponding to the child node contained in the non-terminal node X2 of the index tree, The normalizing the first evaluation value, the second evaluation value, and the third evaluation value comprises: Among them, Z x is one of the first evaluation value, the second evaluation value and the third evaluation value, Z ′ x For Z x The value after normalization processing, m is the total number of the first evaluation value, the second evaluation value and the third evaluation value, and y is the preset range value.

2. Intelligent information system performance evaluation system, characterized in that: include: An evaluation requirement decomposition module (1) is used to obtain evaluation requirements and gradually decompose the evaluation requirements to obtain an indicator hierarchical structure; wherein the indicator hierarchical structure includes a plurality of hierarchical indicators and an inclusion relationship between the hierarchical indicators; An indicator tree generation module (2) is used to screen the hierarchical indicators, use the screened hierarchical indicators as first target indicators, and create an indicator tree according to the first target indicators; wherein the indicator tree includes an inclusion relationship between the first target indicators and the first target indicators; An indicator weight assignment module (3), used for assigning a corresponding indicator weight to each node of the indicator tree; A first evaluation module (4), used to generate a first evaluation questionnaire according to the terminal node of the indicator tree, and to perform evaluation according to the first evaluation questionnaire to obtain the evaluation value corresponding to the terminal node of the indicator tree; A first evaluation value generating module (5), used for calculating a first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value; A second evaluation value generating module (6) is used to call a scale generating interface to generate an evaluation scale, and to perform evaluation according to the evaluation scale to obtain a second target indicator and a second evaluation value corresponding to the second target indicator; A third evaluation value generating module (7) is used to obtain multimodal data, and determine a third target indicator and a third evaluation value corresponding to the third target indicator according to the multimodal evaluation data, wherein the multimodal evaluation data is generated by a mathematical prototype, a VR prototype, a physical prototype or a real prototype of the evaluated system; An evaluation tree generation module (8) is used to construct an evaluation tree based on the evaluation requirements and according to the first target indicator, the second target indicator and the third target indicator corresponding to the root node of the indicator tree; A normalization processing module (9), used for performing normalization processing on the first evaluation value, the second evaluation value and the third evaluation value to obtain corresponding first evaluation value, second evaluation value and third evaluation value; An evaluation weight assigning module (10) is used to assign a corresponding evaluation weight to each node of the evaluation tree; A validity value generating module (11) is used to calculate the validity value of each node of the evaluation tree according to the first evaluation value, the second evaluation value, the third evaluation value and the evaluation weight; an effectiveness evaluation report generating module (12), configured to generate an effectiveness evaluation report according to the evaluation tree and the effectiveness value corresponding to each node of the evaluation tree, The step of calculating the first evaluation value corresponding to each node of the indicator tree according to the indicator tree, the indicator weight and the evaluation value includes: For any terminal node X1 of the index tree, S1=W*X Wherein, S1 is the first evaluation value corresponding to the terminal node X1 of the indicator tree, W is the indicator weight corresponding to the terminal node X1 of the indicator tree, and X is the evaluation value corresponding to the terminal node X1 of the indicator tree; For any non-terminal node X2 of the index tree, Wherein, S2 is the first evaluation value corresponding to the non-terminal node X2 of the index tree, n is the number of child nodes contained in the non-terminal node X2 of the index tree, and W i is the index weight corresponding to the child node contained in the non-terminal node X2 of the index tree, X i is the first evaluation value corresponding to the child node contained in the non-terminal node X2 of the index tree, The normalizing the first evaluation value, the second evaluation value, and the third evaluation value comprises: Among them, Z x is one of the first evaluation value, the second evaluation value and the third evaluation value, Z ′ x For Z x The value after normalization processing, m is the total number of the first evaluation value, the second evaluation value and the third evaluation value, and y is the preset range value.

3. A computing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the method according to claim 1 when executing the computer program.

4. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to claim 1.

Images