A comprehensive performance and capability evaluation method based on system multi-index joint simulation

Abstract

This invention belongs to the field of system evaluation technology and discloses a comprehensive performance evaluation method based on multi-index joint simulation of a system. The method involves dividing the target system into multiple subsystems, constructing a performance and index system for each subsystem including all its performance parameters, performing joint simulation on the target system to obtain simulation data for each performance parameter, decomposing the simulation data into each subsystem, analyzing the correlation between all performance parameters of any subsystem, and comprehensively weighting the performance parameters of any subsystem based on the degree of influence to obtain the performance and evaluation value of any subsystem. This performance and evaluation value of any subsystem is then used as a component performance index of the target system. Finally, the component performance indexes of the target system are comprehensively weighted based on the degree of influence to obtain the comprehensive performance and evaluation value of the target system. This invention can evaluate the comprehensive application performance and efficiency of complex systems.

Description

Technical Field

[0001] This invention belongs to the field of system evaluation technology, specifically relating to a comprehensive performance and effectiveness evaluation method based on multi-index joint simulation of a system. Background Technology

[0002] With the rapid development of computer technology, simulation technology has become an important tool for studying complex systems. In practical applications, system simulation can more comprehensively reflect the overall performance of a system, providing a more accurate basis for performance evaluation. However, the simulation process generates massive amounts of data, characterized by diversity, large data volume, and high-speed generation. How to effectively process and utilize this data for performance evaluation has become an urgent problem to be solved in the field of simulation. Furthermore, there are differences between domestic and international research on performance evaluation using simulation big data. International research started earlier and has developed relatively mature theories and methods, such as statistical performance evaluation and machine learning-based performance evaluation. While domestic research started later, it has developed rapidly in recent years and has achieved some important results.

[0003] Therefore, it is of great significance to provide a comprehensive performance evaluation method based on multi-index joint simulation of the system to evaluate the comprehensive application effectiveness and performance of complex systems and improve the accuracy and efficiency of performance evaluation of complex systems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a comprehensive performance and efficiency evaluation method based on multi-index joint simulation of the system. This method includes the following steps:

[0005] Step 1: Divide the target system into multiple subsystems and construct the performance and efficiency index system for each subsystem. The performance and efficiency index system of any subsystem includes all performance parameters of that subsystem.

[0006] Step 2: Use simulation technology to perform joint simulation and deduction of the target system, and obtain simulation data of each performance parameter;

[0007] Step 3: Classify the simulation data and decompose it into each subsystem;

[0008] Step 4: Perform correlation analysis on all performance parameters of any subsystem, and combine the influence of any performance parameter to perform comprehensive weighting on the performance parameters of any subsystem to obtain the efficiency and performance evaluation value of any subsystem.

[0009] Step 5: Take the performance and evaluation value of any subsystem as the sub-performance index of the target system, and combine the influence of any sub-performance index to perform a comprehensive weighted processing on the sub-performance index of the target system to obtain the comprehensive performance and evaluation value of the target system.

[0010] Specifically, in step 1, the target system is divided into multiple subsystems based on its structure and function.

[0011] Specifically, in step 1, the performance and efficiency indicators of any subsystem include: system performance indicators, reliability indicators, availability indicators, security indicators, maintainability indicators, and / or efficiency indicators.

[0012] Specifically, system performance metrics include response time, throughput, and number of concurrent users;

[0013] Reliability metrics include failure rate and mean time to repair (MTBT).

[0014] Availability metrics include system availability and recovery time;

[0015] Security metrics include data breach rate and number of unauthorized access attempts;

[0016] Maintainability metrics include maintenance costs and maintenance cycles;

[0017] Efficiency indicators include resource utilization and energy consumption.

[0018] Specifically, in step 2, after acquiring the simulation data, the simulation data is preprocessed, including:

[0019] Step 21: Clean the simulation data, remove duplicate, invalid and abnormal data, and obtain the first data;

[0020] Step 22: Perform data type conversion, numerical standardization / normalization, and / or encoding on the first data.

[0021] Specifically, step 4 includes:

[0022] Step 41: Based on the correlation analysis, obtain the first weight value of each performance parameter. At the same time, based on the influence analysis, obtain the second weight value of each performance parameter.

[0023] Step 42: Obtain the simulation data sequence for any performance parameter, calculate the standard deviation of the simulation data sequence, and repeat this step until the standard deviation of all performance parameters is obtained.

[0024] Step 43: The first weight value and the second weight value of any performance parameter are weighted and summed to obtain the third weight value of any performance parameter. Then, the effectiveness and performance evaluation value of any subsystem are calculated based on the third weight value and the standard deviation.

[0025] Specifically, in step 4, the method for obtaining the first weight value is as follows:

[0026] A correlation matrix is ​​constructed based on simulation data. Each row of the correlation matrix represents the data sequence of any performance parameter, and each column represents a point in time.

[0027] The grey relational degree calculation formula is used to calculate the correlation degree between each performance parameter and other performance parameters.

[0028] The first weight value of any performance parameter is calculated based on the correlation degree, and the calculation formula is as follows:

[0029] ,

[0030] Among them, w i r is the first weight value of the i-th performance parameter. i Let represent the correlation degree of the i-th performance parameter, where i is a positive integer from 1 to I, and I is the total number of performance parameters for any subsystem.

[0031] Specifically, in step 4, the method for obtaining the second weight value is as follows:

[0032] Construct a multiple regression model based on all performance parameters of any subsystem;

[0033] The model parameters of the multiple regression model are estimated using the least squares method based on simulation data.

[0034] Calculate the influence value of any performance parameter based on the standard deviation and model parameters corresponding to any performance parameter, and define the influence value as the second weight value. Repeat this step until the second weight value of all performance parameters is obtained.

[0035] Specifically, in step 5, the calculation method for the comprehensive effectiveness and performance evaluation value E is as follows:

[0036] ,

[0037] in, E represents the indicator weight. j Let j be the j-th sub-item performance index, where j is a positive integer from 1 to J, and J is the total number of subsystems.

[0038] Specifically, the weight of any sub-performance indicator is calculated by combining subjective and objective weighting.

[0039] This invention discloses a comprehensive performance evaluation method based on multi-index joint simulation of a system, with at least the following beneficial effects:

[0040] Considering the hierarchical subdivision of indicators, the target system is divided into multiple subsystems, and an efficiency and performance indicator system for each subsystem is constructed. By performing correlation analysis and comprehensive weighting processing on various indicators in the complex system, the performance and efficiency status of each subsystem is obtained. Then, the performance and efficiency status of each subsystem are used as the sub-performance indicators of the entire system, and comprehensive weighting processing is performed again to finally obtain the comprehensive efficiency and performance evaluation results of the entire system. This can comprehensively evaluate the system's performance in various aspects and provide a more comprehensive system performance evaluation.

[0041] By using applications to perform joint simulations and online recording calculations of various indicators in complex systems and their subsystems, the actual operation of the system can be simulated more accurately, and simulation data of various performance parameters can be obtained. These data are closer to the actual operation, improving the accuracy of the evaluation. At the same time, by preprocessing, analyzing correlations, and performing comprehensive weighting on the simulation data, the weight and influence of each performance parameter can be determined based on data-driven methods, making the decision-making process more scientific and objective. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a flowchart of a comprehensive performance evaluation method based on multi-index joint simulation of a system, according to the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the specific embodiments described herein are merely illustrative of the invention and represent only a portion, not all, of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0045] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0046] Figure 1 The diagram shows a flowchart of an embodiment of a comprehensive performance evaluation method based on multi-index joint simulation of a system, provided by the present invention. The flowchart specifically includes the following steps:

[0047] Step 1: Divide the target system into multiple subsystems and construct the performance and efficiency index system for each subsystem. The performance and efficiency index system of any subsystem includes all performance parameters of that subsystem.

[0048] Specifically, in step 1, the target system is divided into multiple subsystems based on its structure and function.

[0049] When partitioning a system, it can be done by dividing it according to its functional modules, with each subsystem responsible for one or more specific functions; or, considering the physical structure of the system, the system can be decomposed into physically independent parts, with each part serving as a subsystem.

[0050] Specifically, in step 1, the performance and efficiency indicators of any subsystem include: system performance indicators, reliability indicators, availability indicators, security indicators, maintainability indicators, and / or efficiency indicators.

[0051] The performance and efficiency indicators of any subsystem include one or more of the above indicators.

[0052] Specifically, system performance metrics include response time, throughput, and number of concurrent users;

[0053] Reliability metrics include failure rate and mean time to repair (MTBT).

[0054] Availability metrics include system availability and recovery time;

[0055] Security metrics include data breach rate and number of unauthorized access attempts;

[0056] Maintainability metrics include maintenance costs and maintenance cycles;

[0057] Efficiency indicators include resource utilization and energy consumption.

[0058] Dividing the system into subsystems based on the structure and function of the target system ensures that the indicator system of each subsystem is closely related to its actual function, thereby improving the relevance and accuracy of the evaluation. At the same time, the effectiveness and performance indicators of any subsystem cover multiple dimensions such as system performance, reliability, availability, security, maintainability, and efficiency, ensuring the comprehensiveness and detail of the evaluation results.

[0059] Step 2: Use simulation technology to perform joint simulation and deduction of the target system, and obtain simulation data of each performance parameter.

[0060] By using an application program to perform joint simulation and online recording of various indicators in complex systems and their subsystems, the simulation platform is selected according to the specific needs and characteristics of the target system, so as to achieve the purpose of joint simulation and simulation of complex systems and ensure the accuracy and efficiency of the simulation.

[0061] Before conducting the simulation, set the simulation objectives, which include the performance metrics to be evaluated, simulation accuracy, and simulation time.

[0062] Specifically, in step 2, after acquiring the simulation data, the simulation data is preprocessed, including:

[0063] Step 21: Clean the simulation data to remove duplicate, invalid, and abnormal data, and obtain the first data.

[0064] Step 22: Perform data type conversion, numerical standardization / normalization, and / or encoding on the first data.

[0065] The data obtained from simulation runs have various characteristics such as different types, different units, and different formats. It is necessary to convert the simulation data into a format suitable for analysis to provide a high-quality dataset for subsequent data analysis and model building.

[0066] Step 3: Classify the simulation data and decompose it into each subsystem.

[0067] Step 4: Perform correlation analysis on all performance parameters of any subsystem, and combine the influence of any performance parameter to perform comprehensive weighting on the performance parameters of any subsystem to obtain the efficiency and performance evaluation value of any subsystem.

[0068] Conducting correlation analysis helps reveal the intrinsic relationships between different performance parameters, contributing to a deeper understanding of the subsystem's behavior and dynamic characteristics. The degree of influence of a performance parameter refers to the magnitude of its impact on the subsystem's performance or effectiveness evaluation results. Calculating the subsystem's effectiveness and performance evaluation values ​​based on correlation analysis and the degree of influence of performance parameters ensures that the evaluation results accurately reflect the actual contribution and importance of each performance parameter to the system's effectiveness.

[0069] Specifically, step 4 includes:

[0070] Step 41: Based on the correlation analysis, obtain the first weight value of each performance parameter. At the same time, based on the influence analysis, obtain the second weight value of each performance parameter.

[0071] Step 42: Obtain the simulation data sequence for any performance parameter, calculate the standard deviation of the simulation data sequence, and repeat this step until the standard deviation of all performance parameters is obtained.

[0072] Step 43: The first weight value and the second weight value of any performance parameter are weighted and summed to obtain the third weight value of any performance parameter. Then, the effectiveness and performance evaluation value of any subsystem are calculated based on the third weight value and the standard deviation.

[0073] For example, the formula for calculating the effectiveness and performance evaluation value of any subsystem is as follows:

[0074] ,

[0075] Where E1 is the performance evaluation value of any subsystem, β1 and β2 are parameter weighting coefficients, W1 is the first weight value, W2 is the second weight value, and SD i S is the current measured value of the i-th performance parameter. i Let i be the standard value of the i-th performance parameter. Let be the standard deviation of the i-th performance parameter.

[0076] Specifically, in step 4, the method for obtaining the first weight value is as follows:

[0077] A correlation matrix is ​​constructed based on simulation data. Each row of the correlation matrix represents the data sequence of any performance parameter, and each column represents a point in time.

[0078] The grey relational degree calculation formula is used to calculate the correlation degree between each performance parameter and other performance parameters.

[0079] The first weight value of any performance parameter is calculated based on the correlation degree, and the calculation formula is as follows:

[0080] ,

[0081] Among them, w i r is the first weight value of the i-th performance parameter. i Let represent the correlation degree of the i-th performance parameter, where i is a positive integer from 1 to I, and I is the total number of performance parameters for any subsystem.

[0082] The correlation between any performance parameter and other performance parameters refers to the measure of the similarity or dissimilarity of their changing trends in a grey system. The magnitude of the correlation reflects the degree of synchronous change between the two parameters, i.e., the consistency of their changes. If the changing trends of the two parameters are consistent, that is, their degree of synchronous change is high, then their correlation is high; conversely, if the changing trends are inconsistent, the correlation is low. Determining the weight of each performance parameter based on correlation analysis provides a scientific basis for system performance evaluation.

[0083] For example, statistical methods (such as correlation analysis, regression analysis) or machine learning techniques (such as decision trees, random forests) can also be used to identify the relationships between performance parameters.

[0084] Specifically, in step 4, the method for obtaining the second weight value is as follows:

[0085] Construct a multiple regression model based on all performance parameters of any subsystem;

[0086] The model parameters of the multiple regression model are estimated using the least squares method based on simulation data.

[0087] Calculate the influence value of any performance parameter based on the standard deviation and model parameters corresponding to any performance parameter, and define the influence value as the second weight value. Repeat this step until the second weight value of all performance parameters is obtained.

[0088] A multiple regression model is constructed using the efficiency and performance evaluation value of any subsystem as the dependent variable and all performance parameters corresponding to any subsystem as independent variables. The least squares method is used to obtain the weight coefficients of each independent variable (i.e., the model parameters of any subsystem).

[0089] Preferably, the formula for calculating the influence value of any performance parameter is as follows:

[0090] ,

[0091] Among them, w j Let j be the influence value of the j-th performance parameter. Let C be the standard deviation of the j-th performance parameter. j Let be the model parameter of the j-th performance parameter, where j is a positive integer from 1 to J, and J is the total number of performance parameters for any subsystem.

[0092] By calculating the standard deviation of a performance parameter, its volatility or dispersion can be measured. A larger standard deviation indicates greater variability and potentially a greater impact on system effectiveness and performance. Combining this with parameters from a multiple regression model (such as regression coefficients) indicates the strength and direction of the correlation between variables, allowing for further quantification of the contribution or influence of each performance parameter on the overall system effectiveness and performance. According to the technical solution of this invention, it helps to more accurately understand the role of each performance parameter in the system, more precisely estimate the impact of performance parameters on system effectiveness, and improve the accuracy of the assessment.

[0093] Step 5: Take the performance and evaluation value of any subsystem as the sub-performance index of the target system, and combine the influence of any sub-performance index to perform a comprehensive weighted processing on the sub-performance index of the target system to obtain the comprehensive performance and evaluation value of the target system.

[0094] Specifically, in step 5, the calculation method for the comprehensive effectiveness and performance evaluation value E is as follows:

[0095] ,

[0096] in, Let Ej be the index weight, where Ej is the j-th sub-item performance index, j is a positive integer from 1 to J, and J is the total number of subsystems.

[0097] Specifically, the weight of any sub-performance indicator is calculated by combining subjective and objective weighting.

[0098] The weight of any of the above-mentioned performance indicators represents the degree of influence of that performance indicator on the target system. Preferably, a regression model can be constructed, using standardized coefficients (Beta coefficients) to measure the degree of influence of each performance indicator on the effectiveness of the target system.

[0099] Preferably, the overall performance and performance evaluation value of the target system can be calculated according to the method and steps in step 4 for calculating the performance and performance evaluation value of any subsystem.

[0100] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0101] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0102] The above-described embodiments merely illustrate preferred embodiments of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A comprehensive performance evaluation method based on multi-index joint simulation of a system, characterized in that, Includes the following steps: Step 1: Divide the target system into multiple subsystems and construct the performance and efficiency index system for each subsystem. The performance and efficiency index system of any subsystem includes all performance parameters of any subsystem. Step 2: Use simulation technology to perform joint simulation and deduction on the target system, and obtain simulation data for each performance parameter; Step 3: Classify the simulation data and decompose it into each of the subsystems; Step 4: Perform correlation analysis on all performance parameters of any subsystem, and combine the influence of any performance parameter to perform comprehensive weighting on the performance parameters of any subsystem to obtain the efficiency and performance evaluation value of any subsystem. Step 5: Take the efficiency and performance evaluation value of any of the subsystems as the sub-performance indicators of the target system, and combine the influence of any sub-performance indicator to perform a comprehensive weighted processing on the sub-performance indicators of the target system to obtain the comprehensive efficiency and performance evaluation value of the target system. Step 4 includes: Step 41: Based on the correlation analysis, obtain the first weight value of all the performance parameters respectively; at the same time, based on the influence degree analysis, obtain the second weight value of all the performance parameters respectively. Step 42: Obtain the simulation data sequence of any of the performance parameters, calculate the standard deviation of the simulation data sequence, and repeat this step until the standard deviation of all the performance parameters is obtained. Step 43: The first weight value and the second weight value of any performance parameter are weighted and summed to obtain the third weight value of any performance parameter. Then, the efficiency and performance evaluation value of any subsystem are calculated based on the third weight value and the standard deviation. The method for obtaining the first weight value is as follows: A correlation matrix is ​​constructed based on the simulation data. Each row of the correlation matrix is ​​a data sequence for any performance parameter, and each column represents a point in time. The grey relational degree calculation formula is used to calculate the correlation degree between each performance parameter and other performance parameters. The first weight value of any performance parameter is calculated based on the correlation, and the calculation formula is as follows: ， Among them, w i r is the first weight value of the i-th performance parameter. i Let be the correlation degree of the i-th performance parameter, where i is a positive integer from 1 to I, and I is the total number of performance parameters of any of the subsystems.

2. The method according to claim 1, characterized in that, In step 1, the target system is divided into multiple subsystems based on its structure and function.

3. The method according to claim 1, characterized in that, In step 1, the performance and efficiency indicators of any of the subsystems include: system performance indicators, reliability indicators, availability indicators, security indicators, maintainability indicators, and / or efficiency indicators.

4. The method according to claim 3, characterized in that, The system performance metrics include response time, throughput, and number of concurrent users; The reliability metrics include failure rate and mean time to repair. The availability metrics include system availability and recovery time; The security metrics include data breach rate and number of unauthorized accesses; The maintainability indicators include maintenance costs and maintenance cycles; The efficiency indicators include resource utilization rate and energy consumption.

5. The method according to claim 1, characterized in that, In step 2, after acquiring the simulation data, the simulation data is preprocessed, and the preprocessing includes: Step 21: Clean the simulation data to remove duplicate, invalid, and abnormal data, and obtain the first data; Step 22: Perform data type conversion, numerical standardization / normalization, and / or encoding on the first data.

6. The method according to claim 1, characterized in that, In step 4, the method for obtaining the second weight value is as follows: Construct a multiple regression model based on all the performance parameters of any of the subsystems; The model parameters of the multiple regression model are estimated using the least squares method based on the simulation data. The influence value of any performance parameter is calculated based on the standard deviation and the model parameter corresponding to any performance parameter. The influence value is defined as the second weight value. This step is repeated until the second weight value of all performance parameters is obtained.

7. The method according to claim 1, characterized in that, In step 5, the method for calculating the comprehensive effectiveness and performance evaluation value E is as follows: ， in, E represents the indicator weight. j Let j be the j-th sub-performance index, where j is a positive integer from 1 to J, and J is the total number of the subsystems.

8. The method according to claim 7, characterized in that, The weight of any of the aforementioned sub-performance indicators is calculated by combining subjective and objective weighting.

Images