Simulation experiment efficiency evaluation index analysis method and device based on confidence coefficient

Abstract

The invention relates to a confidence-based simulation experiment efficiency evaluation index analysis method and device. The method comprises the following steps: according to existing historical data and expert experience in a complex system simulation experiment scene, determining an initial threshold interval of an efficiency evaluation index and calculating a confidence coefficient; judging whether the confidence of the initial threshold interval reaches the standard or not, and if yes, verifying the credibility of the efficiency evaluation result based on the initial threshold interval; if the result does not reach the standard, a Bayesian posterior algorithm is adopted to carry out threshold interval iteration updating, the confidence degree of the updated threshold interval is calculated based on a normal distribution inverse CDF function until the confidence degree reaches the standard, and reliability verification of the efficiency evaluation result is carried out based on the current threshold interval. According to the method, scientific quantification of the efficiency evaluation index confidence and automatic updating of the index threshold interval based on the confidence can be realized in a complex system simulation experiment scene with limited samples and cognitive uncertainty, and the credibility of the efficiency evaluation result can be improved.

Description

Technical Field

[0001] This application relates to the technical field of performance evaluation, and particularly to a method and device for analyzing performance evaluation indicators of simulation experiments based on confidence. Background Art

[0002] Performance evaluation technology is widely used in fields such as complex system simulation experiments, equipment evaluation, and scheme optimization. With the rapid development of simulation technology, performance evaluation based on simulation experiments can provide key support for the design, development, testing, and evaluation of equipment in real complex system scenarios. However, existing research mostly focuses on performance evaluation methods and applications themselves, lacking systematic verification of the credibility of performance evaluation results, making it difficult to determine whether the comprehensive performance evaluation of equipment by simulation experiments is accurate and effective.

[0003] For performance evaluation results, they mainly come from the statistical calculation of performance evaluation indicators. Therefore, the credibility of performance evaluation results can be evaluated by conducting an analysis of the trustworthiness of the reliability and accuracy of performance evaluation indicators.

[0004] However, the analysis of the trustworthiness of traditional performance evaluation indicators mainly uses probability to study uncertainty problems and quantify the impact of uncertainty on reliability. This usually requires a large amount of sample data to ensure the accuracy of the results. However, in the scenario of complex system simulation experiments, obtaining a large amount of high-quality data is often costly or infeasible, resulting in a reduction in the credibility of performance evaluation results. On the other hand, in complex system simulation experiments, epistemic uncertainty often dominates, and traditional probability statistics methods are also difficult to fully reflect the true uncertainty of the system. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a method and device for analyzing performance evaluation indicators of simulation experiments based on confidence, breaking through the limitations of the traditional probability framework, and realizing the scientific quantification of the confidence of performance evaluation indicators in the complex system simulation experiment scenario of limited samples and epistemic uncertainty, so as to improve the credibility of performance evaluation results.

[0006] A method for analyzing performance evaluation indicators of simulation experiments based on confidence, the method comprising: Determine the initial threshold interval of the performance evaluation indicator and perform confidence calculation according to the existing historical data and expert experience in the complex system simulation experiment scenario; Determine whether the confidence level of the initial threshold interval reaches a preset confidence level threshold. If it reaches, directly adopt the effectiveness evaluation index based on this initial threshold interval to verify the credibility of the effectiveness evaluation result; if it does not reach, use the Bayesian posterior algorithm to iteratively update the threshold interval of the effectiveness evaluation index, and calculate the confidence level of the updated threshold interval of the effectiveness evaluation index based on the inverse CDF function of the normal distribution until the confidence level of the updated threshold interval reaches the preset confidence level threshold, and adopt the effectiveness evaluation index based on the current threshold interval to verify the credibility of the effectiveness evaluation result.

[0007] In one embodiment, according to the historical data and expert experience in the complex system simulation experiment scenario, determine the initial threshold interval of the effectiveness evaluation index and calculate the confidence level, including: According to the historical data and expert experience in the complex system simulation experiment scenario, determine the initial threshold interval of the effectiveness evaluation index, expressed as ; where and respectively represent the lower limit and the upper limit of the initial threshold interval; According to the historical data or simulation experiment data in the complex system simulation experiment scenario, perform confidence level calculation on , expressed as: ; where is the confidence level of the initial threshold interval, , is the index mean in the historical data or simulation experiment data, is the sample standard deviation of the historical data or simulation experiment data, and , is the th data in the historical data or simulation experiment data, and , is the total number of data in the historical data or simulation experiment data.

[0008] In one embodiment, the confidence level threshold is set to 95%. If the confidence level of the threshold interval of the effectiveness evaluation index is greater than or equal to 95%, it means that the probability that this threshold interval covers the true experimental result of the effectiveness evaluation index is greater than or equal to 95%. Directly adopt the effectiveness evaluation index based on the current threshold interval to verify the credibility of the effectiveness evaluation result; otherwise, use the Bayesian posterior algorithm to iteratively update the threshold interval of the effectiveness evaluation index until the confidence level of the threshold interval is greater than or equal to 95%.

[0009] In one embodiment, use the Bayesian posterior algorithm to iteratively update the threshold interval of the effectiveness evaluation index, including: First, determine the prior parameters of the threshold interval according to the threshold interval updated in the previous round of performance evaluation indicators. Then, calculate based on the prior parameters combined with the new simulation experiment data in the complex system simulation experiment scenario to obtain the posterior parameters of the threshold interval. Finally, update the threshold interval updated in the previous round according to the posterior parameters to obtain the threshold interval updated in this round of performance evaluation indicators. Among them, when initially updating the threshold interval of the performance evaluation indicator, the threshold interval updated in the previous round is the initial threshold interval.

[0010] In one of the embodiments, initially updating the threshold interval of the performance evaluation indicator includes the following steps: Establish a normal prior distribution for the initial threshold interval of the performance evaluation indicator to determine the prior parameters of the threshold interval. Specifically, set the lower limit initial value and the upper limit initial value to be and , respectively, and determine the prior parameters of the threshold interval based on the lower and upper limit initial values. Among them, the prior parameters include the mean location and the variance estimate , which are respectively expressed as: ; ; Among them, is the quantile of the standard normal distribution, α is the significance level, and when , ; is the expert credibility attenuation factor; and respectively represent the lower and upper limits of the initial threshold interval; Conduct a simulation experiment in the complex system simulation experiment scenario to obtain n independent simulation experiment data, and calculate based on the simulation experiment data and the prior parameters to determine the posterior parameters of the threshold interval, including and , which are respectively expressed as: ; ; Among them, represents the i th simulation experiment data; and respectively represent the posterior mean location parameter and the posterior variance estimate parameter; Update the initial threshold interval according to the posterior parameters to obtain the updated threshold interval of the performance evaluation indicator, which is expressed as: ; ; in, and Respectively represent the updated threshold interval lower and upper limits.

[0011] In one embodiment, calculating the confidence of the updated threshold interval of the performance evaluation indicator based on the normal distribution inverse CDF function includes: Based on the normal distribution inverse CDF function, calculate the updated threshold interval of the performance evaluation index Confidence , expressed as: ; in, is the inverse CDF function of the standard normal distribution.

[0012] In one of the embodiments, the method further includes: setting a threshold for the number of iterative updates of the threshold interval; if the number of iterative updates exceeds the threshold, and the confidence of the updated threshold interval still does not reach the confidence threshold, resetting the initial threshold interval of the performance evaluation indicator, and based on a feedback mechanism of the confidence level on the threshold interval, re-iteratively updating the threshold interval of the performance evaluation indicator.

[0013] A confidence-based simulation experiment effectiveness evaluation index analysis device, the device comprising: The initial calculation module is used to determine the initial threshold range of the performance evaluation index and perform confidence calculation based on the existing historical data and expert experience in the complex system simulation experiment scenario; The confidence-based iterative update module is used to determine whether the confidence of the initial threshold interval reaches the preset confidence threshold. If it reaches it, the performance evaluation index based on the initial threshold interval is directly adopted to verify the credibility of the performance evaluation result; if it does not reach it, the threshold interval of the performance evaluation index is iteratively updated by using the Bayesian posterior algorithm, and the confidence of the threshold interval after the performance evaluation index is updated is calculated based on the inverse CDF function of the normal distribution, until the confidence of the updated threshold interval reaches the preset confidence threshold, and the performance evaluation index based on the current threshold interval is adopted to verify the credibility of the performance evaluation result.

[0014] The above method and device for analyzing the effectiveness evaluation index of simulation experiments based on confidence constructs a feedback mechanism of confidence for the threshold interval of the effectiveness evaluation index. By combining expert experience and statistical data to quantify the confidence of the threshold interval, it can more comprehensively reflect the true uncertainty of complex system simulation experiment scenarios. When the confidence level does not meet the standard, the Bayesian posterior algorithm is used to automatically update and correct the threshold interval, achieving online incremental learning. It can directly judge and confirm whether the threshold interval of the index is reliable through the automatic iterative update of the threshold interval and the recalculation of confidence under the condition of limited samples and cognitive uncertainty. When the confidence level meets the standard, the effectiveness evaluation index based on the current threshold interval is directly adopted to verify the credibility of the effectiveness evaluation result, thereby improving the credibility of the effectiveness evaluation result.

[0015] Furthermore, in the automatic iterative update process of the threshold interval of the present application, Bayesian prior parameters are adopted, which can effectively avoid the excessive oscillation of the updated calculation result of the threshold interval caused by limited samples, thereby reducing the iterative calculation period of the index threshold interval and improving the update calculation efficiency of the index threshold interval. Brief Description of the Drawings

[0016] Figure 1 It is a schematic flowchart of the method for analyzing the effectiveness evaluation index of simulation experiments based on confidence in an embodiment; Figure 2 It is a schematic structural diagram of the device for analyzing the effectiveness evaluation index of simulation experiments based on confidence in an embodiment. Detailed Embodiments

[0017] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] In one embodiment, as Figure 1 shown, a method for analyzing the effectiveness evaluation index of simulation experiments based on confidence is provided, including the following steps: Step S1, determine the initial threshold interval of the effectiveness evaluation index and calculate the confidence based on the existing historical data and expert experience in the complex system simulation experiment scenario.

[0019] Step S2, determine whether the confidence of the initial threshold interval reaches the preset confidence threshold. If it reaches, directly adopt the effectiveness evaluation index based on the initial threshold interval to verify the credibility of the effectiveness evaluation result; if it does not reach, proceed to step S3.

[0020] Step S3: Update the threshold interval of the performance evaluation index using the Bayesian posterior algorithm, and calculate the confidence level of the updated threshold interval of the performance evaluation index based on the inverse CDF (Cumulative Distribution Function) function of the normal distribution.

[0021] Step S4: Further determine whether the confidence level of the updated threshold interval reaches the preset confidence level threshold. If it reaches, adopt the performance evaluation index based on the current threshold interval to verify the credibility of the performance evaluation result; if it does not reach, repeat Step S3 for iterative update of the threshold interval until the confidence level of the updated threshold interval reaches the preset confidence level threshold.

[0022] In one embodiment, based on the existing historical data and expert experience in the complex system simulation experiment scenario, determine the initial threshold interval of the performance evaluation index and calculate the confidence level, including: Based on the existing historical data and expert experience in the complex system simulation experiment scenario, determine the initial threshold interval of the performance evaluation index, expressed as ; where and respectively represent the lower and upper limits of the initial threshold interval; Based on the historical data or simulation experiment data in the complex system simulation experiment scenario, perform confidence level calculation on , expressed as: ; where is the confidence level of the initial threshold interval, , is the index mean in the historical data or simulation experiment data, is the sample standard deviation of the historical data or simulation experiment data, and , is the th data in the historical data or simulation experiment data, and , is the total number of data in the historical data or simulation experiment data.

[0023] By combining expert experience and statistical data for confidence level quantification calculation, on the one hand, expert experience can quantify the cognitive uncertainty in the complex system simulation experiment scenario (such as decision rules, blue side strategies and other factors that are difficult to statistically), and statistical data (such as historical data and simulation experiment data) can quantify the random uncertainty in the complex system simulation experiment scenario (such as equipment performance fluctuations, environmental noise). The combination of the two can more comprehensively reflect the true uncertainty of the complex system simulation experiment scenario and improve the reliability of confidence level calculation.

[0024] In one of the embodiments, the confidence threshold is set to 95%. If the confidence level of the threshold interval of the performance evaluation index is greater than or equal to 95%, it means that the probability that the threshold interval covers the true experimental result of the performance evaluation index is greater than or equal to 95%. The confidence of the performance evaluation result is directly verified by adopting the performance evaluation index based on the current threshold interval; otherwise, it means that the probability that the threshold interval covers the true experimental result of the performance evaluation index is relatively low, affecting the credibility of the performance evaluation index result. It is necessary to further adjust the threshold interval of the index. Specifically, the Bayesian posterior algorithm is used to iteratively update the threshold interval of the performance evaluation index until the confidence level of the threshold interval is greater than or equal to 95%.

[0025] Specifically, using the Bayesian posterior algorithm to iteratively update the threshold interval of the performance evaluation index includes: first, determining the prior parameters of the threshold interval according to the threshold interval updated in the previous round of the performance evaluation index, then calculating according to the prior parameters combined with the new simulation experiment data in the complex system simulation experiment scenario to obtain the posterior parameters of the threshold interval, and finally updating the threshold interval updated in the previous round according to the posterior parameters to obtain the threshold interval updated in this round of the performance evaluation index; among them, when initially updating the threshold interval of the performance evaluation index, the threshold interval updated in the previous round is the initial threshold interval.

[0026] The initial update of the threshold interval of the performance evaluation index includes the following steps: First, establish a normal prior distribution for the initial threshold interval of the performance evaluation index to determine the prior parameters of the threshold interval; specifically, set the lower limit initial value and the upper limit initial value of the threshold interval to be and respectively, and determine the prior parameters of the threshold interval based on the lower and upper limit initial values; among them, the prior parameters include the mean positioning and the variance estimation , which are respectively expressed as: ; ; where is the quantile of the standard normal distribution , α is the significance level, and when , ; is the expert credibility decay factor, and the empirical value is taken as 1.2 - 1.5; and respectively represent the lower and upper limits of the initial threshold interval;

[0027] Then, conduct a simulation experiment in the complex system simulation experiment scenario to obtain nIndependent simulation experiment data, and calculate based on the simulation experiment data and prior parameters to determine the posterior parameters of the threshold interval, including and , which are respectively expressed as: ; ; Among them, represents the i th simulation experiment data; and respectively represent the posterior mean localization parameter and the posterior variance estimation parameter; Finally, update the initial threshold interval according to the posterior parameters to obtain the threshold interval of the updated effectiveness evaluation index , which is expressed as: ; ; Among them, and respectively represent the lower and upper limits of the updated threshold interval .

[0028] Thus, based on the inverse CDF function of the normal distribution, calculate the confidence level of the threshold interval of the updated effectiveness evaluation index , which is expressed as: ; Among them, is the inverse CDF function of the standard normal distribution.

[0029] Specifically, if , then the confidence level of the current threshold interval meets the requirements, and the threshold interval of the effectiveness evaluation index can be determined according to , and the credibility verification of the effectiveness evaluation result is carried out by adopting the effectiveness evaluation index based on this threshold interval. If , then the confidence level of the current threshold interval still does not meet the requirements, and it is necessary to perform iterative update according to until the confidence level .

[0030] Furthermore, to prevent the confidence level after the adjustment of the threshold interval from not meeting the requirements all the time, this method also includes: setting the iteration update times threshold of the threshold interval. If the iteration update times exceed this threshold and the confidence level of the updated threshold interval still does not reach the confidence level threshold, reset the initial threshold interval of the effectiveness evaluation index, and based on the feedback mechanism of the confidence level for the threshold interval, re-perform the iterative update of the threshold interval of the effectiveness evaluation index.

[0031] Furthermore, this method can iteratively update the threshold intervals of single or multiple performance evaluation indicators based on confidence, including performance evaluation indicators such as maneuvering distance, range, equipment utilization rate, and anti-interference ability in complex system simulation experiment scenarios.

[0032] In summary, a method for analyzing performance evaluation indicators of simulation experiments based on confidence provided by this application can more comprehensively reflect the true uncertainty of complex system simulation experiment scenarios by combining expert experience and statistical data to quantify the confidence of threshold intervals. When the confidence does not meet the standard, the Bayesian posterior algorithm is used to automatically update and correct the threshold intervals, achieving online incremental learning. It can directly judge and confirm whether the threshold intervals of indicators are reliable through automatic iterative update of threshold intervals and recalculation of confidence under the condition of limited samples and cognitive uncertainty. When the confidence meets the standard, the performance evaluation indicators based on the current threshold intervals are directly adopted to verify the credibility of performance evaluation results, thereby improving the credibility of performance evaluation results.

[0033] In one embodiment, as Figure 2 shown, a device for analyzing performance evaluation indicators of simulation experiments based on confidence is provided, including: An initial calculation module 201, configured to determine an initial threshold interval of a performance evaluation indicator and calculate the confidence based on historical data and expert experience existing in a complex system simulation experiment scenario.

[0034] An iterative update module 202 based on confidence is configured to determine whether the confidence of the initial threshold interval reaches a preset confidence threshold. If it reaches, directly adopt the performance evaluation indicator based on this initial threshold interval to verify the credibility of the performance evaluation result; if it does not reach, use the Bayesian posterior algorithm to iteratively update the threshold interval of the performance evaluation indicator, and calculate the confidence of the updated threshold interval of the performance evaluation indicator based on the inverse CDF function of the normal distribution until the confidence of the updated threshold interval reaches the preset confidence threshold, and adopt the performance evaluation indicator based on the current threshold interval to verify the credibility of the performance evaluation result.

[0035] For the specific limitations of the device for analyzing performance evaluation indicators of simulation experiments based on confidence, reference can be made to the limitations of the method for analyzing performance evaluation indicators of simulation experiments based on confidence in the above text, which will not be elaborated here. Each module in the above device for analyzing performance evaluation indicators of simulation experiments based on confidence can be implemented in whole or in part through software, hardware, and their combinations. The above modules can be embedded in the processor of a computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above respective modules.

[0036] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0037] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for analyzing the effectiveness evaluation index of simulation experiments based on confidence, characterized in that, The method includes: Determining an initial threshold interval of the effectiveness evaluation index and calculating the confidence level according to the existing historical data and expert experience in the complex system simulation experiment scenario; Judging whether the confidence level of the initial threshold interval reaches a preset confidence level threshold. If it reaches, directly adopt the effectiveness evaluation index based on the initial threshold interval to verify the credibility of the effectiveness evaluation result; if not, use the Bayesian posterior algorithm to iteratively update the threshold interval of the effectiveness evaluation index, and calculate the confidence level of the updated threshold interval of the effectiveness evaluation index based on the inverse CDF function of the normal distribution until the confidence level of the updated threshold interval reaches the preset confidence level threshold, and adopt the effectiveness evaluation index based on the current threshold interval to verify the credibility of the effectiveness evaluation result.

2. The method according to claim 1, characterized in that, Determining an initial threshold interval of the effectiveness evaluation index and calculating the confidence level according to the existing historical data and expert experience in the complex system simulation experiment scenario, including: Based on the existing historical data and expert experience in the complex system simulation experiment scenario, determine the initial threshold interval of the effectiveness evaluation index, denoted as ; where and represent the lower and upper limits of the initial threshold interval respectively; Based on the historical data or simulation experiment data in the complex system simulation experiment scenario, for perform confidence calculation, expressed as: ； Among them, is the confidence level of the initial threshold interval, , is the mean value of the indicators in the historical data or simulation experiment data, is the sample standard deviation of the historical data or simulation experiment data, and , is the th data in the historical data or simulation experiment data, and , is the total number of data in the historical data or simulation experiment data.

3. The method according to claim 1, characterized in that, The confidence level threshold is set to 95%. If the confidence level of the threshold interval of the effectiveness evaluation index is greater than or equal to 95%, it means that the probability that the threshold interval covers the true experimental result of the effectiveness evaluation index is greater than or equal to 95%, and directly adopt the effectiveness evaluation index based on the current threshold interval to verify the credibility of the effectiveness evaluation result; otherwise, use the Bayesian posterior algorithm to iteratively update the threshold interval of the effectiveness evaluation index until the confidence level of the threshold interval is greater than or equal to 95%.

4. The method according to claim 3, wherein Using the Bayesian posterior algorithm to iteratively update the threshold interval of the effectiveness evaluation index, including: First, determining the prior parameters of the threshold interval according to the threshold interval updated in the previous round of the effectiveness evaluation index, then calculating according to the prior parameters combined with the new simulation experiment data in the complex system simulation experiment scenario to obtain the posterior parameters of the threshold interval, and finally updating the threshold interval updated in the previous round according to the posterior parameters to obtain the threshold interval updated in this round of the effectiveness evaluation index; where, when initially updating the threshold interval of the effectiveness evaluation index, the threshold interval updated in the previous round is the initial threshold interval.

5. The method according to claim 4, wherein The initial update of the threshold interval of the effectiveness evaluation index includes the following steps: By establishing a normal prior distribution for the initial threshold interval of the performance evaluation index, the prior parameters of the threshold interval are determined; specifically, the lower limit initial value of the threshold interval is set and the upper limit initial value are respectively and , and the prior parameters of the threshold interval are determined based on the lower and upper limit initial values; wherein, the prior parameters include the mean positioning and the variance estimation , which are respectively expressed as: ； ； Among them, is the standard normal distribution quantile, α is the significance level, and when ; is the expert credibility attenuation factor; and respectively represent the lower and upper limits of the initial threshold interval; Performing a simulation experiment in a complex system simulation experiment scenario to obtain n independent sets of simulation experiment data, and calculating based on the simulation experiment data and the prior parameters to determine the posterior parameters of the threshold interval, including and , which are respectively expressed as: ； ； Among them, represents the i th simulation experiment data; and respectively represent the posterior mean positioning parameter and the posterior variance estimation parameter; Update the initial threshold interval according to the posterior parameter to obtain the threshold interval after the performance evaluation index is updated , which is expressed as: ； ； Among them, and respectively represent the lower limit and the upper limit of the updated threshold interval .

6. The method according to claim 5, characterized in that Calculating the confidence level of the updated threshold interval of the effectiveness evaluation index based on the inverse CDF function of the normal distribution, including: Calculate the threshold interval after updating the performance evaluation index based on the inverse CDF function of the normal distribution confidence level which is expressed as: ； wherein, is the inverse CDF function of the standard normal distribution.

7. The method according to claim 1, characterized in that, The method further includes: Setting an iteration update times threshold of the threshold interval. If the confidence level of the updated threshold interval still does not reach the confidence level threshold when the iteration update times exceed this threshold, reset the initial threshold interval of the effectiveness evaluation index, and based on the feedback mechanism of the confidence level for the threshold interval, re-perform the iterative update of the threshold interval of the effectiveness evaluation index.

8. An analysis device for evaluating the effectiveness of simulation experiments based on confidence, characterized in that, The device includes: An initial calculation module, configured to determine an initial threshold interval of the effectiveness evaluation index and calculate the confidence level according to the existing historical data and expert experience in the complex system simulation experiment scenario; A confidence-based iterative update module is used to determine whether the confidence of the initial threshold interval reaches a preset confidence threshold. If it reaches, the effectiveness evaluation index based on this initial threshold interval is directly adopted to verify the credibility of the effectiveness evaluation result; if it does not reach, the Bayesian posterior algorithm is used to iteratively update the threshold interval of the effectiveness evaluation index, and the confidence of the threshold interval of the effectiveness evaluation index after update is calculated based on the inverse CDF function of the normal distribution until the confidence of the updated threshold interval reaches the preset confidence threshold, and the effectiveness evaluation index based on the current threshold interval is adopted to verify the credibility of the effectiveness evaluation result.

Images