A method for evaluating the performance efficiency of an information system

Abstract

The present invention discloses a method for evaluating the performance efficiency of an information system. The evaluation method includes step 1: establishing an evaluation index system for the performance efficiency of an information system; step 2: calculating the subjective and objective weights of each evaluation index in the evaluation index system; step 3: using the Pearson correlation coefficient method to calculate the degree of synergy between the subjective and objective weights; according to the degree of synergy, selecting synergistic combination weighting or discrete combination weighting to obtain the optimal solution for the evaluation weights; step 4: constructing an information system performance efficiency evaluation model: combining the obtained evaluation weights to establish an intuitive and fuzzy two-dimensional evaluation model of the information system performance efficiency evaluation index. The present invention can provide a more scientific approach to the performance efficiency evaluation of information systems, so as to enhance the application of information system performance efficiency evaluation in normalized testing.

Description

Technical Field

[0001] The invention relates to a method for evaluating the performance efficiency of an information system. Background Art

[0002] Information system performance efficiency testing refers to the use of performance efficiency testing tools (such as LoadRunner, JMeter, etc.) to simulate the normal, abnormal and peak load states of the information system to the maximum extent, test performance efficiency indicators such as average response time, average processor occupancy, user access volume, analyze and evaluate indicators, and determine the degree to which the information system meets performance efficiency requirements.

[0003] In the national standard GB / T25000.10-2016, "Systems and Software Engineering - System and Software Quality Requirements and Evaluation (SQuaRE) - Part 10: System and Software Quality Model," performance efficiency is classified as one of the eight characteristics for evaluating software quality. In 2021, performance efficiency testing was separately written into a national standard—GB / T39788-2021, "Performance Testing Methods for Systems and Software Engineering." This standard document details the performance efficiency quality measurement indicators, testing process, and requirements model for large-scale information systems. The importance of information system performance efficiency testing is self-evident. Based on the evaluation results of information system performance efficiency testing, the execution efficiency, resource utilization, and system capacity of the information system are evaluated; abnormal states of indicators in the evaluation information are analyzed to locate system bottlenecks such as the time characteristics of performance efficiency, resource utilization, and capacity, providing a basis for improving the performance efficiency of the information system; based on the evaluation information, multiple tests, adjustments, and verification of the optimal configuration of the information system are carried out to achieve information system optimization. Therefore, evaluating the performance efficiency of information systems is of great significance.

[0004] However, there are still many problems in the current information system performance efficiency evaluation: (1) The evaluation model is simple. The existing information system performance efficiency test model relies too much on the professional knowledge and experience of experts, or simply linearly weights objective test data. The model is simple and the evaluation results are not reasonable and scientific. (2) The display of information system evaluation results is single and lacks the description of evaluation results from multiple perspectives such as intuitive and fuzzy, which may lead to incomplete expression of evaluation information. These defects limit the application of information system performance efficiency evaluation in normalized testing. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the prior art, the main purpose of the present invention is to provide an evaluation method for the performance efficiency of an information system, to provide a more scientific approach for the performance efficiency evaluation of an information system, and to enhance the application of the performance efficiency evaluation of an information system in normalized testing.

[0006] The main objectives of the present invention are achieved through the following technical measures.

[0007] A method for evaluating the performance efficiency of an information system, comprising:

[0008] Step 1: Establish an evaluation index system for information system performance efficiency;

[0009] 1.1) Determination of the evaluation index set for information system performance efficiency;

[0010] 1.2) Normalization of evaluation data in the evaluation index set: Forward normalization of reverse data, peak data, and interval data to standardize the data properties of the index data so that the larger the index value, the better the performance;

[0011] 1.3) Standardization of evaluation data in the evaluation indicator set: After discretization of data and optimization of the model, the improved power coefficient method is used to standardize the evaluation data to the same order of magnitude;

[0012] Step 2: Calculate the subjective and objective weights of each evaluation indicator in the evaluation indicator system;

[0013] Step 3: Use the Pearson correlation coefficient method to calculate the degree of synergy between subjective and objective weights. Based on the degree of synergy, select synergistic combination weighting or discrete combination weighting to find the optimal solution for evaluation weights.

[0014] Step 4: Constructing an information system performance efficiency evaluation model: Based on the obtained evaluation weights, establish an intuitive and fuzzy dual-dimensional evaluation model for the information system performance efficiency evaluation indicators; 4.1) Use numerical display, weight ratio display, and evaluation comparison display to directly display the evaluation results to intuitively and objectively reflect the comprehensive evaluation results of the information system performance efficiency; 4.2) Apply cloud model theory to describe the comprehensive qualitative evaluation of the performance efficiency evaluation numerical conversion in natural language, and fuzzily display the comprehensive evaluation results of performance efficiency.

[0015] In some embodiments, step 4.1 specifically includes the following:

[0016] 4.1.1) Numerical Display: Using a linear weighted synthesis method, combined with the evaluation weight system of the optimal solution, the quantitative data of the test results are linearly weighted and combined to form an evaluation value that intuitively reflects the comprehensive performance and efficiency evaluation results;

[0017] 4.1.2) Weight and proportion display: Use pie charts, doughnut charts and other types of visual graphics to intuitively display the indicator weight and proportion of the total;

[0018] 4.1.3) Evaluation Comparison Display: Use comparative visualizations such as radar charts, bar charts, and indicator cards to intuitively compare multi-metric evaluation data and highlight outliers between them, thereby locating system performance bottlenecks and optimizing system performance.

[0019] In some embodiments, step 4.2 specifically includes the following:

[0020] 4.2.1) Construction of a Fuzzy Evaluation Set: Based on the degree of performance efficiency met by the information system, a fuzzy evaluation set is constructed, describing the comprehensive performance efficiency score in natural language. The evaluation values of the indicators in the evaluation set are divided into multiple level intervals. The cloud characteristic calculation formula is then used to calculate the cloud model characteristic values for each level interval.

[0021] 4.2.2) Drawing of Standard Model: Based on the expected value, entropy, and super entropy of the hierarchical interval cloud model eigenvalues, a forward cloud generator is used to map qualitative concepts to characteristic cloud droplets. Based on the coordinate values of the characteristic cloud droplets, a standard model of information system performance efficiency is drawn as a measurement scale for evaluation;

[0022] 4.2.3) Generation of fuzzy evaluation results: The evaluation data is calculated using an inverse cloud generator and converted into a cloud model based on the final combined weights. The fuzziness and stability of the evaluation are determined based on entropy and hyperentropy. Based on the comparison of expected values with the standard model, a comprehensive qualitative evaluation of the performance efficiency evaluation numerical conversion is described in natural language in combination with the fuzzy evaluation set to generate the evaluation results.

[0023] In some embodiments, the fuzzy evaluation set constructed in step 4.2.1) is as follows:

[0024]

[0025] Among them, Ex is the expected value, En is the entropy, and He is the excess entropy.

[0026] In some embodiments, step 1.1) specifically includes the following:

[0027] 1.1.1) Using time characteristics, resource utilization, capacity, and performance efficiency compliance as primary indicators, and refining at least some of these primary indicators into multiple representative secondary indicators;

[0028] Among them, the secondary indicators of time characteristics include average response time, response time sufficiency, average turnaround time, turnaround time sufficiency, and average throughput; the secondary indicators of resource utilization include average processor occupancy, average memory occupancy, average I / O device occupancy, and bandwidth occupancy; the secondary indicators of capacity include transaction processing capacity, user access volume, and sufficiency of user access growth;

[0029] 1.1.2) Supplementation of specific evaluation indicators: To measure the processing capacity and stability of information systems, two secondary indicators are added to the primary evaluation indicator of time characteristics: the transaction success rate of completing defined transactions per unit time and the timeout error rate of transactions that fail due to timeouts or internal system errors.

[0030] In some embodiments, step 1.2) normalizing the evaluation data specifically includes the following:

[0031] 1.2.1) For reverse data such as average response time, response time adequacy, average turnaround time, and timeout error rate, the reciprocal unification method is used to complete positive normalization;

[0032] 1.2.2) For peak data such as average I / O device occupancy and bandwidth occupancy, set the optimal occupancy rate as the peak value, and use the binary function method to derive, decompose, and calculate it to achieve positive normalization;

[0033] 1.2.3) For interval data such as average processor occupancy and average memory occupancy, the optimal interval is selected as the peak interval, and the interval data is reduced using a discretization equation to complete positive normalization.

[0034] In some embodiments, step 1.3, normalization of the evaluation data specifically includes the following:

[0035] 1.3.1) Discretize the evaluation index data into a fixed standardized area based on the expected target points;

[0036] 1.3.2) Map the fixed standardized area to different levels and correspond to different efficacy coefficient values [G min , G h , G y , G max ];

[0037] 1.3.3) Set standard values, combine the efficacy coefficient values of different grades, and use the improved efficacy coefficient method to calculate the evaluation data score based on the expected target point. This will convert the evaluation data of different indicators to the same order of magnitude, overcoming the problem of inconsistent characteristics such as dimensions and orders of magnitude. The calculation formula of the improved efficacy coefficient value is as follows:

[0038]

[0039] In the formula, y is the index data to be standardized, [I min , I h , I y , I max ] is the normalized indicator data.

[0040] In some embodiments, step 2 specifically includes the following:

[0041] 2.1) Complex performance efficiency evaluation indicators are hierarchically and structured. Using the analytic hierarchy process (AHP), a hierarchical indicator structure consisting of a target layer, a criterion layer, and an indicator layer is constructed to obtain subjective evaluation weights.

[0042] 2.2) Use the entropy weight method to objectively calculate the probability of standardized historical evaluation data, the information entropy and information utility value of the estimated indicators, and obtain the objective evaluation weight after normalization;

[0043] 2.3) There are many evaluation indicators based on performance efficiency. The three-scaling method is used to optimize the obtained subjective evaluation weights to reduce the judgment value and simplify the calculation process.

[0044] In some embodiments, step 3 specifically includes the following:

[0045] 3.1) Construct a weight matrix and use the covariance method to calculate the error of the weight population in the weight matrix. The positive and negative values of the covariance results indicate the positive and negative correlation between the weights.

[0046] 3.2) Use the Pearson correlation coefficient method to calculate the degree of synergy between subjective and objective weights;

[0047] 3.3) Based on the results of the synergy between weights, select the optimal combination weighting method to obtain the evaluation weight;

[0048] If the synergy between subjective and objective weights is strong, the multiplication fusion normalization algorithm is used to create a synergistic combination weighting calculation;

[0049] If the synergy between subjective and objective weights is weak, the contrast and conflict of the CRITIC method is used to construct a discrete combination weighting calculation.

[0050] In some embodiments, in step 3.3), when the synergy correlation coefficient is in the range of [0.7, 1], a multiplication fusion normalization algorithm is used to create a synergistic combination weighting calculation; otherwise, a discrete combination weighting calculation is constructed using the contrast and conflict of the CRITIC method.

[0051] The present invention has the following beneficial effects:

[0052] The present invention constructs an information system performance efficiency evaluation index based on time characteristics, resource utilization, capacity and performance efficiency compliance, and normalizes the index data in a positive manner and adopts an improved efficacy coefficient method; calculates subjective and objective weights, and adopts a multiplication fusion normalization algorithm to create a collaborative combination weighting, constructs a discrete combination weighting based on the contrast and conflict of the CRITIC method, calculates the synergy according to the Pearson correlation coefficient method, and selects the optimal combination weighting; uses numerical values, weight proportions, and evaluation comparisons to intuitively display the evaluation results, applies cloud model theory to fuzzily display the comprehensive evaluation of performance efficiency, and completes the intuitive and fuzzy dual-dimensional display of the evaluation results; with the rapid advancement of the scale of informatization, the emergence of big data, diversified, and high-concurrency information systems has higher requirements for the evaluation of performance efficiency. The present invention provides a more scientific approach for the performance efficiency evaluation of information systems, which will help to enhance the application and promotion of the performance efficiency evaluation of information systems.

[0053] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of an embodiment of the evaluation method of the present invention;

[0055] Figure 2 is a framework diagram of an evaluation index set in an embodiment of the present invention;

[0056] Figure 3 is a schematic diagram of the hierarchical division of performance efficiency evaluation indicators in an embodiment of the present invention;

[0057] Figure 4 4-1 is a schematic diagram showing the weight ratio in an embodiment of the present invention; wherein, 4-1 is the average response time, 4-2 is the sufficiency of the response time, 4-3 is the average turnaround time, 4-4 is the sufficiency of the turnaround time, 4-5 is the average throughput, 4-6 is the transaction success rate, 4-7 is the timeout error rate, 4-8 is the average processor occupancy, 4-9 is the average memory occupancy, 4-10 is the average I / O device occupancy, 4-11 is the bandwidth occupancy, 4-12 is the transaction processing capacity, 4-13 is the user access volume, 4-14 is the sufficiency of the user access volume, and 4-15 is the compliance of performance efficiency;

[0058] Figure 5 5-1 represents time characteristics, 5-2 represents performance efficiency compliance, 5-3 represents capacity, and 5-4 represents resource utilization.

[0059] Figure 66-1 represents extremely low performance (very poor), 6-2 represents low performance (poor), 6-3 represents moderate performance (qualified), and 6-4 represents high performance (excellent).

[0060] Figure 7 7-1 represents extremely low performance (very poor), 7-2 represents low performance (poor), 7-3 represents moderate performance (qualified), 7-4 represents high performance (excellent), and 7-5 represents the cloud model of the evaluation data. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and implementation examples. It should be understood that the specific implementation examples described herein are merely for explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0062] like Figure 1 As shown, the information system performance efficiency evaluation method provided by the present invention includes the following steps:

[0063] Step 1: Establish an evaluation index system for information system performance efficiency;

[0064] 1.1) Determination of the evaluation indicator set for information system performance efficiency: In order to enhance the representativeness, authority and accessibility of the indicators, the evaluation indicator set is determined based on national standards, industry standards, enterprise standards and other standard documents.

[0065] 1.1.1) Based on the national standard GB / T25000.51-2016, time characteristics, resource utilization, capacity and performance efficiency compliance are used as first-level indicators, and at least some of the first-level indicators are refined into multiple representative second-level indicators; such as Figure 2 As shown, the secondary indicators are determined based on the national standard documents GB / T25000.23-2019 and GB / T39788-2021 as follows:

[0066] Secondary indicators of time characteristics include average response time, adequacy of response time, average turnaround time, adequacy of turnaround time, and average throughput;

[0067] Secondary indicators of resource utilization include average processor occupancy, average memory occupancy, average I / O device occupancy, and bandwidth occupancy;

[0068] Secondary indicators of capacity include transaction processing capacity, user access volume, and adequacy of user access growth;

[0069] 1.1.2) Supplementing Specific Evaluation Indicators: To measure the processing capacity and stability of information systems, the first-level evaluation indicator of time characteristics is supplemented with two secondary indicators: the transaction success rate (the rate at which defined transactions are completed per unit time) and the timeout error rate (the rate at which transactions fail due to timeouts or internal system errors).

[0070] Specifically, for the transaction success rate, multiple measurements are taken, and after removing abnormal data, the average value is taken. The calculation formula is as follows:

[0071]

[0072] Where Ts is the number of successful transactions, Tf is the number of failed transactions, and n1 is the number of tests.

[0073] The timeout error rate is calculated as follows:

[0074]

[0075] Among them E over Indicates the number of timeout errors, T sum Represents the total number of transactions, and n2 represents the number of tests.

[0076] 1.2) Normalization of evaluation data in the evaluation indicator set: Use performance testing tools (such as LoadRunner and JMeter) to obtain test data for each evaluation indicator. Given the characteristic of test data that performance increases with increasing indicator values, forward normalization is performed on the reverse data, peak data, and interval data. This standardizes the data properties of the indicator data to the point where the larger the indicator value, the better the performance. This overcomes the computational difficulties caused by the varying growth properties of the evaluation indicators. The details are as follows:

[0077] 1.2.1) For inverse data such as average response time, response time adequacy, average turnaround time, and timeout error rate (the smaller the indicator data, the better the performance efficiency requirements), the reciprocal unification method is used to complete positive normalization;

[0078] 1.2.2) For peak data such as average I / O device utilization and bandwidth utilization (which exhibit characteristics of peak data types: too low a value results in wasted resources, while too high a value results in decreased performance), the optimal utilization rate is set as the peak value. This is then derived, decomposed, and calculated using a binary function method to achieve positive normalization. For example, if the peak average I / O device utilization is 35% and the peak bandwidth utilization is 30%, the calculation results are:

[0079]

[0080] Among them, Mmax Indicates the maximum distance between the indicator value and the peak value, calculated as follows:

[0081] M max =Max{|D i -D best |} (Formula 4)

[0082] Among them, the i-th data of the peak data type indicator sequence is Di, and the peak value is D best , the maximum value after normalization is N.

[0083] 1.2.3) For interval data such as average processor occupancy and average memory occupancy (with fixed intervals where system resources are fully utilized and performance efficiency is optimal), select the optimal interval as the peak interval, and use the discretization equation to reduce the interval data to complete positive normalization. For example, when the average processor occupancy is between 65% and 70%, the processor resources are fully utilized and performance efficiency is optimal; select the optimal interval and use it as the peak interval [L peak , H peak ], and its calculation formula is:

[0084]

[0085] Among them, M n For: M n =max{L peak -min{D i},max{D i}-H peak}

[0086] Where N is the maximum value after normalization, D i The i-th data in the interval data type indicator sequence.

[0087] 1.3) Standardization of evaluation data in the evaluation index set: After discretization of data and optimization of modeling, the improved power coefficient method is used to standardize the evaluation data to the same order of magnitude.

[0088] Specifically, the normalized data still needs to overcome the problem of inconsistent characteristics such as dimensions and magnitudes. The improved power coefficient method is used. After discretization of the data and optimization of the model, the data is normalized to the same magnitude to facilitate the calculation of the subsequent comprehensive evaluation. The details are as follows:

[0089] 1.3.1) Discretize the evaluation indicator data into a fixed standardized area;

[0090] Considering that the normalization results of different sample data are not comparable, according to various types of standard documents, a fixed indicator value that meets the performance requirements is selected as the constraint critical value [L min, L max ], select the qualified point L of the expected target h And excellent point L y For discrete points, discretize the evaluation data into a fixed standardized area, such as the constraint critical value of the average response time [L min , L max ]=[1.5,12], indicating that the average response time is 1.5 seconds for the best performance efficiency, and 12 seconds is an unacceptable critical value. The expected target qualified point L h And excellent point L y 8 seconds and 3 seconds respectively, where [L min , L y , L h , L max ]=[1.5,3,8,12] After normalization by the reciprocal unification method, the result is [I min , I h , I y , I max ]=[1 / 12, 1 / 8, 1 / 3, 1 / 1.5].

[0091] 1.3.2) Map the fixed standardized area to different levels and correspond to different efficacy coefficient values [G min , G h , G y , G max ], such as the efficacy coefficient value of the average response time [G min , G h , G y , G max ]=[0.2, 0.7, 0.9, 1].

[0092] 1.3.3) Set standard values, combine the efficacy coefficient values of different levels, and use the improved efficacy coefficient method to calculate the score of the evaluation data based on the expected target point.

[0093] The traditional efficacy coefficient method directly uses linear mapping to normalize the index data, lacks constraints on scores at different levels, and cannot reflect the scores of qualified and excellent performance efficiency levels. Based on the traditional efficacy coefficient method, the expected target points (qualified points, excellent points) and constraint critical values are set (as shown in Table 1), and the performance efficiency index data are discretized into different fixed standardized areas, mapped to different levels, and corresponding to different efficacy coefficient values one by one. Combined with the standard value, the improved efficacy coefficient method is used to calculate the scores of evaluation index data at different levels to adapt to multiple expected goals; among them, the calculation formula of the improved efficacy coefficient value is as follows: As shown, the evaluation data of different indicators are converted to the same order of magnitude, overcoming the problem of inconsistent characteristics such as dimension and order of magnitude; for example, the average response time of the test is 2 seconds, which is normalized to 1 / 2 by the reciprocal consistency method, which is less than the constraint point (Imax=1) but greater than the excellent point (Iy=1 / 3), and its efficacy coefficient is 0.9, which is calculated by the following formula 6: (1 / 2-1 / 3) / (1 / 1.5-1 / 3)*(1-0.9)+0.9, and the result is 0.95; in particular, as shown in Table 2, the normalized data Data is used to obtain the standardized result Bdata by the improved efficacy coefficient method.

[0094]

[0095] Where y is the evaluation index data to be standardized, [I min , I h , I y , I max ] is the normalized indicator data.

[0096] Table 1 Correspondence table of expected target points and efficacy coefficient values of indicators

[0097]

[0098] Table 2 Standardization cases of efficacy coefficient method

[0099]

[0100] Step 2: Calculate the subjective and objective weights of each evaluation indicator in the evaluation indicator system. To improve the rationality of the weights, the hierarchical analysis method is used to obtain the subjective evaluation weights; to strengthen the scientificity and rigor of the evaluation weights, the entropy weight method is used to calculate the objective data evaluation weights. The details are as follows:

[0101] 2.1) Using the analytic hierarchy process to obtain subjective evaluation weights;

[0102] In order to hierarchically and structure the complex performance efficiency evaluation indicators, such as time characteristics, resource utilization, capacity, and performance efficiency compliance, they are divided into a hierarchical structure;

[0103] Using the hierarchical analysis method, a hierarchical index structure including the target layer, the criterion layer and the index layer is constructed (such as Figure 3 As shown in Figure 7), the expert professional knowledge and experience are comprehensively considered to subjectively construct a judgment matrix. After consistency verification, the subjective evaluation weights are obtained (as shown in the following formula 7) to improve the rationality of the weights.

[0104]

[0105] Among them, Mat ij is the value of the i-th row and j-th column of the judgment matrix.

[0106] 2.2) Use the entropy weight method to objectively calculate the probability of standardized historical evaluation data, the information entropy and information utility value of the estimated indicators, and obtain the objective evaluation weight after normalization;

[0107] In order to avoid the situation where the subjective preference of weight calculation is too strong, the entropy weight method is used to objectively calculate the probability P of the standardized historical evaluation data based on the principle that the smaller the variance of the data, the greater the information entropy, which means the corresponding weight value is smaller. ij , calculate the information entropy E of the index i The weight of the indicators after normalization is calculated by combining the information utility value and objectively calculating the evaluation weight to improve the scientificity and rigor of the evaluation weight. The calculation formula for the objective data evaluation weight is:

[0108]

[0109] Among them, DW i Refers to the weight of the i-th indicator, E i is the information entropy of indicator i; E i The calculation formula is:

[0110]

[0111] Where n3 represents the number of weights.

[0112] 2.3) The obtained subjective evaluation weights are optimized using the three-scaling method.

[0113] Due to the large number of performance efficiency evaluation indicators, the above-mentioned judgment matrix construction has the problems of arbitrariness and large computational complexity. The three-scaling method is used to optimize the obtained subjective evaluation weights, reduce the judgment values, simplify the calculation process, and overcome the problem of deviation in consistency verification results caused by complex calculations. The final subjective and objective evaluation weights are shown in Table 3 below.

[0114] Table 3 Subjective and objective evaluation weights

[0115]

[0116] Step 3: Use the Pearson correlation coefficient method to calculate the degree of synergy between subjective and objective weights; based on the degree of synergy, select synergistic combination weighting or discrete combination weighting to obtain the optimal solution for the evaluation weights.

[0117] To elaborate, performance efficiency indicators vary in nature, and subjective and objective weight assignments differ. To measure the consistency of synergy between weights, a weight matrix is constructed, and the covariance method is used to represent the positive and negative correlations between weights. The Pearson correlation coefficient method is used to calculate the degree of synergy between weights and verify the consistency of subjective and objective weight change trends. The details are as follows:

[0118] 3.1) Construct a weight matrix and use the covariance method to calculate the error of the weight population in the weight matrix. The positive and negative values of the covariance results indicate the positive and negative correlation between the weights.

[0119] A set of real numbers arranged in a rectangular array is used to store subjective and objective weights such as subjective evaluation weights and objective data evaluation weights, and a weight matrix U = [W1, W2, ..., W n ]=[[SW1,SW2,...,SW n ], [DW1, DW2, ..., DW n ],...,[XW1,XW2,...,XW n ]], facilitating effective numerical analysis between weights.

[0120] In order to measure the correlation between subjective and objective weights, the covariance method is used to calculate the error of the weight population in the weight matrix. The positive and negative values of the covariance indicate the positive and negative correlation between the weights. A positive value indicates that the weights are positively correlated, and a negative value indicates that the weights are negatively correlated. The covariance formula is as follows:

[0121]

[0122] where x i , x j Respectively represent the weights of the i-th and j-th rows in the weight matrix U, x ik represents the kth weight value in the i-th row of the weight matrix U, xj k Represents the kth weight value in the jth row of the weight matrix U.

[0123] 3.2) Use the Pearson correlation coefficient method to calculate the degree of synergy between subjective and objective weights;

[0124] In order to measure the degree of association between weights, the Pearson correlation coefficient method is used. The coefficient is calculated by dividing the covariance representing the correlation between weights by the standard deviation of the weights. The numerical value of the coefficient is used to represent the degree to which the weights meet the synergy, and to verify the consistency of the subjective and objective weight change trends.

[0125] The closer the Pearson correlation coefficient value is to 1, the stronger the synergy between the weights and the more consistent the change trend. The calculation formula of the Pearson correlation coefficient method is as follows:

[0126]

[0127] 3.3) Based on the results of the synergy between weights, select the optimal combination weighting to obtain the evaluation weight;

[0128] In order to comprehensively consider the rationality, scientificity and rigor of subjective and objective evaluation weights, a combined weighting method is adopted to organically combine subjective and objective weights; collaborative combined weighting and discrete combined weighting are set, and the optimal combined weighting is selected based on the verification results of the synergy between weights. The details are as follows:

[0129] If the synergy between subjective and objective weights is strong, that is, when the synergy correlation coefficient is in the range of [0.7, 1], and the weight importance is similar, the multiplication fusion normalization algorithm is used to create a synergistic combination weighting calculation; the ratio of the product of the combination weights of the weighting method to the sum of the product of the combination weights is calculated to evenly distribute the subjective and objective weights.

[0130] Among them, for A subjective and objective weighting methods, the combined weight TWi of the i-th indicator calculated by the multiplication fusion normalization algorithm is:

[0131]

[0132] The combined weight product of the i-th indicator using the a-th weighting method is The result of adding the combined weight products of each indicator using the ath weighting method is

[0133] If the synergy between subjective and objective weights is weak, that is, the synergy correlation coefficient is not in the range of [0.7, 1], and the weight difference is large, the contrast and conflict of the CRITIC method are used to construct a discrete combination weighting calculation. The formula is as follows:

[0134] ZW i =α1W1+α2W2+…+α n W n (Formula 13)

[0135] Among them, the weight coefficient meets the constraints:

[0136] α1+α2+...+α n =1 (Formula 14)

[0137] The CRITIC method is used to measure the gap and conflict degree between weight values based on the contrast and conflict of weight data. The standard deviation is used to calculate the size of the gap between weight values. The larger the standard deviation, the larger the gap between values. Finally, the weight coefficient is calculated. The calculation process is as follows:

[0138] I. Calculate the contrast and conflict of weight data. The conflict of weights is expressed as follows:

[0139]

[0140] Where r kj is the correlation coefficient between indicators t and j.

[0141] II. Set the amount of information to an intermediate value, organically combine contrast and conflict, and comprehensively measure the relative importance of weights. The greater the amount of information, the greater the importance of the weight. The information calculation formula is as follows:

[0142]

[0143] Where η j Indicates the contrast quantitative value.

[0144] III. Calculate the weight coefficient. Based on the comparability and conflict of weight data, the CRITIC method is used to calculate the weight coefficient as follows:

[0145]

[0146] Among them, the greater the importance of the weight, the greater its weight coefficient.

[0147] The Pearson correlation coefficient method was used to calculate the degree of synergy between the subjective and objective weights in Table 3 using Formula 11, and the Pearson correlation coefficient Pearson = 0.565 was obtained, indicating that the synergy between the weights was weak. Since the synergy correlation coefficient was not in the range of [0.7, 1], discrete combination weighting was selected to solve the final evaluation weight to ensure the optimization of the performance efficiency index weight combination. According to Formula 17, the subjective evaluation weight coefficient = 0.425 and the objective evaluation weight coefficient = 0.575 were calculated. According to Formula 13, the optimal combination weight ZW = [0.091, 0.084, 0.081, 0.078, 0.075, 0.071, 0.069, 0.065, 0.068, 0.060, 0.064, 0.057, 0.053, 0.046, 0.038].

[0148] Step 4: Constructing the information system performance efficiency evaluation model: Based on the obtained evaluation weights, establish an intuitive and fuzzy dual-dimensional evaluation model for the information system performance efficiency evaluation indicators;

[0149] To elaborate, we take the CRM information management system of a large domestic industrial robot manufacturing company as an example, and use numerical display, weight ratio display, and evaluation comparison display to directly display the evaluation results, so as to intuitively and objectively reflect the comprehensive evaluation results of the information system performance efficiency; using cloud model theory, we use natural language to describe the comprehensive qualitative evaluation of the performance efficiency evaluation numerical conversion, and fuzzy display the comprehensive evaluation results of performance efficiency. The details are as follows:

[0150] 4.1) Directly display the evaluation results using numerical values, weighted percentages, and comparative evaluations to intuitively and objectively reflect the comprehensive evaluation results of information system performance and efficiency;

[0151] In order to intuitively and objectively reflect the comprehensive evaluation results, the linear weighted comprehensive method is used to calculate the comprehensive evaluation value, and the proportion visualization diagram is used to intuitively represent the relationship between the indicator weight part and the whole; the comparative visualization graphics are used to intuitively compare multiple indicator data and highlight the outliers between multiple indicator data in order to locate the system performance bottleneck.

[0152] 4.1.1) Numerical Presentation: Considering the independence of primary and secondary indicators, a linear weighted synthesis method is used. Combined with the weights calculated by the optimal combination weighting, the quantitative data of the test results are linearly weighted to form an evaluation value. By constructing the primary and secondary indicator data into an evaluation function with a single numerical target, the comprehensive evaluation results are intuitively and objectively reflected. The linear weighted combination evaluation value formula is as follows:

[0153]

[0154] Among them, SW i represents the weight of the i-th indicator in the optimal combination weighted calculation, X i is the test value of the i-th indicator.

[0155] 4.1.2) Weight ratio display: There are many performance efficiency indicators. In order to intuitively display the relative importance of the weight of a single indicator in the comprehensive evaluation, pie charts, donut charts and other visual graphics are used to intuitively show the indicator value and its proportion to the total, such as Figure 4 As shown in the figure, a pie chart is used to visually display the proportion of the indicator to the overall weight, showing the relationship between the part and the whole.

[0156] 4.1.3) Evaluation and Comparison Display: Performance efficiency indicators are clearly layered. To show the size relationship between the same level, use radar charts, bar charts, indicator cards and other comparative visualization graphics to intuitively compare multiple indicator data and highlight outliers between multiple indicator data to locate system performance bottlenecks and achieve system performance tuning, such as Figure 5 As shown in the figure, a radar chart is used to visually compare the evaluation results of time characteristics, resource utilization, capacity and performance efficiency compliance. Figure 5 It can be seen that the time characteristic performance of the information system is the best, and the capacity of the system is relatively poor. The capacity should be increased to improve the performance of the information system.

[0157] 4.2) Using cloud model theory, describe the comprehensive qualitative evaluation of performance efficiency evaluation numerical conversion in natural language, and fuzzily display the comprehensive evaluation results of performance efficiency;

[0158] In order to comprehensively and qualitatively evaluate performance efficiency from numerous indicator systems, cloud model theory is adopted to describe the comprehensive qualitative evaluation of performance efficiency evaluation numerical conversion in natural language. First, a fuzzy evaluation set is constructed, and then a forward cloud generator is used to draw a standard model of performance efficiency as the evaluation scale. Combined with the optimal combination of weighted weights, it is converted into a cloud model, compared with the standard model, and the evaluation results are generated.

[0159] 4.2.1) Construction of a Fuzzy Comment Set: Based on the degree to which the system meets performance efficiency requirements, a fuzzy comment set describing the comprehensive performance efficiency score is constructed. The evaluation values of the indicators in the comment set are divided into multiple level intervals. The cloud feature calculation formula is then used to calculate the cloud model feature values for each level interval. The cloud feature calculation formula is:

[0160]

[0161] Where Ex is the expected value, M max 、M min are the maximum and minimum boundaries respectively, and k is the linear relationship value between entropy En and super entropy He.

[0162] The constructed fuzzy comment set is shown in Table 4 below:

[0163] Table 4 Fuzzy comment set data

[0164]

[0165]

[0166] 4.2.2) Based on the expectation, entropy, and super entropy of the hierarchical interval cloud model eigenvalues, a forward cloud generator is used to map qualitative concepts to characteristic cloud droplets, including the quantitative value of the characteristic cloud droplets and their membership. The calculation formula is:

[0167]

[0168] According to the coordinate values of characteristic cloud droplets (x, y(x)), a standard model of performance efficiency is drawn (such as Figure 6 as a measure of evaluation.

[0169] 4.2.3) Generation of fuzzy evaluation results: Calculate (Ex i ,Eni , He i The results of ) are shown in Table 5 below. The indicator data of the CRM information management system are measured 5 times, and the 5 measurement results are calculated using Formula 21 and converted into cloud model characteristic parameters (as shown in the data processing results in Table 5).

[0170]

[0171] in, S is the mean and variance of the five measurement data.

[0172] Table 5 CRM information management system performance efficiency data

[0173]

[0174] Combined with the weights of the optimal combination, the evaluation result of the overall goal is converted into a cloud model, whose characteristic parameters (Yx, Yn, Yh) = (0.833, 0.022, 0.005), and the (Ex i ,En i , He i ) and the weight of the optimal combination weighting can be obtained.

[0175]

[0176] The characteristic parameters (Yx, Yn, Yh) are plotted into a cloud model and compared with the standard model. The comparison results are as follows: Figure 7 As shown, the cloud model of the evaluation data exhibits a normal distribution and is trending towards stability. Comparing the expected value with the standard model reveals that the evaluation data evaluation set is U3, with an evaluation grade of moderate performance (qualified), essentially meeting the information system's performance efficiency requirements. The probability of performance bottlenecks is low, and the information system is unlikely to crash. Combined with the intuitive display of poor system capacity and the need to increase capacity to improve system performance, it is clear that capacity is a significant factor in lowering the performance of this CRM system. Based on this evaluation, the performance efficiency of the information system is evaluated from both intuitive and fuzzy dimensions.

[0177] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for evaluating the performance efficiency of an information system, comprising: Step 1: Establish an evaluation index system for information system performance efficiency; 1.1) Determine the evaluation index set for information system performance efficiency; 1.2) Normalize the evaluation data within the evaluation index set: Forward normalize the reverse data, peak data, and interval data to standardize the data properties of the index data so that the larger the index value, the better the performance; 1.3) Normalize the evaluation data within the evaluation index set: After discretizing the data and optimizing the model, use the improved power coefficient method to normalize the evaluation data to the same order of magnitude; Step 2: Calculate the subjective and objective weights of each evaluation indicator in the evaluation indicator system; Step 3: Use the Pearson correlation coefficient method to calculate the degree of synergy between subjective and objective weights. Based on the degree of synergy, select synergistic combination weighting or discrete combination weighting to find the optimal solution for evaluation weights. Step 4: Constructing an Information System Performance Efficiency Evaluation Model: Based on the obtained evaluation weights, establish an intuitive and fuzzy dual-dimensional evaluation model for the information system performance efficiency evaluation indicators. 4.1) Directly display the evaluation results using numerical values, weighted percentages, and comparative evaluations to intuitively and objectively reflect the comprehensive evaluation results of the information system's performance efficiency. 4.2) Applying cloud model theory, describe the comprehensive qualitative evaluation of the performance efficiency evaluation numerical conversion in natural language, and fuzzily display the comprehensive evaluation results of performance efficiency. Step 1.3: Standardization of evaluation data includes the following: 1.3.1) Discretize the evaluation indicator data into a fixed standardized area based on the expected target points; 1.3.2) Map the fixed standardized area to different levels and correspond to different efficacy coefficient values [G min , G h , G y , G max ]; 1.3.3) Set standard values, combine the efficacy coefficient values of different grades, and use the improved efficacy coefficient method to calculate the scores of evaluation data based on the expected target point, so that the evaluation data of different indicators are converted to the same order of magnitude, overcoming the problem of inconsistent dimensional and order of magnitude characteristics; Among them, The improved power coefficient value is calculated as follows: ； In the formula, y is the index data to be standardized, [I min , I h , I y , I max ] is the normalized indicator data; Step 2 specifically includes the following: 2.1) Complex performance efficiency evaluation indicators are hierarchically and structured. Using the analytic hierarchy process (AHP), a hierarchical indicator structure consisting of a target layer, a criterion layer, and an indicator layer is constructed to obtain subjective evaluation weights. 2.2) Use the entropy weight method to objectively calculate the probability of standardized historical evaluation data, the information entropy and information utility value of the inferred indicators, and obtain the objective evaluation weight after normalization; 2.3) There are many evaluation indicators based on performance efficiency. The three-scaling method is used to optimize the obtained subjective evaluation weights to reduce the judgment value and simplify the calculation process.

2. The method for evaluating information system performance efficiency according to claim 1, wherein: Step 4.1 specifically includes the following: 4.1.1) Numerical Display: Using a linear weighted synthesis method, combined with the evaluation weight system of the optimal solution, the quantitative data of the test results are linearly weighted and combined to form an evaluation value that intuitively reflects the comprehensive performance and efficiency evaluation results; 4.1.2) Weight and proportion display: Use pie charts, doughnut charts and other types of visual graphics to intuitively display the indicator weight and proportion of the total; 4.1.3) Evaluation Comparison Display: Use comparative visualizations such as radar charts, bar charts, and indicator cards to intuitively compare multi-metric evaluation data and highlight outliers to identify system performance bottlenecks and optimize system performance.

3. The method for evaluating information system performance efficiency according to claim 1, wherein: Step 4.2 specifically includes the following: 4.2.1) Construction of a Fuzzy Evaluation Set: Based on the degree of performance efficiency met by the information system, a fuzzy evaluation set is constructed, describing the comprehensive performance efficiency score in natural language. The evaluation values of the indicators in the evaluation set are divided into multiple level intervals. The cloud characteristic calculation formula is then used to calculate the cloud model characteristic values for each level interval. 4.2.2) Drawing of Standard Model: Based on the expected value, entropy, and superentropy of the hierarchical interval cloud model eigenvalues, a forward cloud generator is used to map qualitative concepts to characteristic cloud droplets. Based on the coordinate values of the characteristic cloud droplets, a standard model of information system performance efficiency is drawn as a measurement scale for evaluation; 4.2.3) Generation of Fuzzy Evaluation Results: Evaluation data is calculated using an inverse cloud generator and converted into a cloud model based on the final combined weights. The fuzziness and stability of the evaluation are determined based on entropy and hyperentropy. A comprehensive qualitative evaluation of the performance efficiency evaluation numerical conversion is performed using natural language descriptions of the fuzzy evaluation set, based on comparisons with the expected value and the standard model, to generate the evaluation results.

4. The method for evaluating information system performance efficiency according to claim 3, wherein: The fuzzy evaluation set constructed in step 4.2.1) is as follows: Among them, Ex is the expected value, En is the entropy, and He is the excess entropy.

5. The method for evaluating information system performance efficiency according to claim 1, wherein: Step 1.1) specifically includes the following: 1.1.1) Take time characteristics, resource utilization, capacity, and performance efficiency compliance as first-level indicators, and refine at least some of these first-level indicators into multiple representative second-level indicators; Among them, the secondary indicators of time characteristics include average response time, response time sufficiency, average turnaround time, turnaround time sufficiency, and average throughput; the secondary indicators of resource utilization include average processor occupancy, average memory occupancy, average I / O device occupancy, and bandwidth occupancy; the secondary indicators of capacity include transaction processing capacity, user access volume, and sufficiency of user access growth; 1.1.2) Supplementation of Specific Evaluation Indicators: To measure the processing capacity and stability of information systems, the first-level evaluation indicator of time characteristics is supplemented with two secondary indicators: the transaction success rate (the rate at which defined transactions are completed per unit time) and the timeout error rate (the rate at which transactions fail due to timeouts or internal system errors).

6. The method for evaluating information system performance efficiency according to claim 5, wherein: Step 1.2) Normalization of evaluation data specifically includes the following: 1.2.1) For inverse data such as average response time, response time adequacy, average turnaround time, and timeout error rate, use the reciprocal unification method to achieve positive normalization; 1.2.2) For peak data such as average I / O device occupancy and bandwidth occupancy, set the optimal occupancy rate as the peak value. Use the binary function method to derive, decompose, and calculate it to achieve positive normalization. 1.2.3) For interval data such as average processor utilization and average memory utilization, select the optimal interval as the peak interval, and use a discretization equation to reduce the interval data to achieve positive normalization.

7. The method for evaluating information system performance efficiency according to claim 1, wherein: Step 3 specifically includes the following: 3.1) Construct a weight matrix and use the covariance method to calculate the error of the weight population in the weight matrix. The positive and negative values of the covariance results indicate the positive and negative correlation between the weights. 3.2) Use the Pearson correlation coefficient method to calculate the degree of synergy between subjective and objective weights; 3.3) Based on the results of the synergy between weights, select the optimal combination weighting method to obtain the evaluation weights; If the synergy between subjective and objective weights is strong, the multiplication fusion normalization algorithm is used to create a synergistic combination weighting calculation; If the synergy between subjective and objective weights is weak, the contrast and conflict of the CRITIC method is used to construct a discrete combination weighting calculation.

8. The method for evaluating information system performance efficiency according to claim 7, wherein: In step 3.3), when the synergy correlation coefficient is in the range of [0.7, 1], the multiplication fusion normalization algorithm is used to create a synergistic combination weighting calculation; otherwise, the contrast and conflict of the CRITIC method are used to construct a discrete combination weighting calculation.

Images