A method and device for performance evaluation of complex systems considering missing data

Abstract

The application discloses a complex system performance evaluation method and device considering missing data, and relates to the technical field of performance evaluation. The method comprises the following steps: calculating an estimated value, data integrity, target weight and reliability of a test index according to observation data of the test index of a complex system; constructing a reference value of an evaluation grade space, and calculating an output utility of the complex system when containing missing data in combination with the estimated value, the target weight and the reliability; and calculating the sensitivity of the complex system according to the output utility and the data integrity, and evaluating the output utility of the complex system by using the sensitivity when the data integrity changes. The application introduces comprehensive indexes such as data integrity, target weight and reliability, can adapt to non-equal-interval characteristics caused by high-frequency and non-uniform-interval testing, reduces the interference of missing key characteristics on system performance evaluation, and dynamically captures the change trend of system output utility through correlation analysis of data integrity and sensitivity.

Description

Technical Field

[0001] This application relates to the field of performance evaluation technology, and more specifically, to a method and apparatus for performance evaluation of complex systems that take into account missing data. Background Technology

[0002] As complex systems evolve towards higher speeds, lighter weights, and greater intelligence, the integrity of their operational data and the reliability of their performance assessments face severe challenges. Improper performance status assessments of complex systems can lead to multiple problems: First, misjudgments of key indicators may trigger cascading failures, such as power grid collapses or large-scale production line downtime. Second, assessment models that do not consider missing data are prone to deviating from actual operating conditions, leading to biased maintenance strategies. Third, inaccurate energy efficiency assessments may trigger invalid dispatch commands, increasing industrial costs. Therefore, constructing a scientific and comprehensive assessment system is crucial for ensuring the safe and efficient operation of complex systems.

[0003] Performance evaluation of complex systems is essentially a systematic analysis method based on multi-dimensional data fusion. It quantifies and evaluates system efficiency and operational quality by analyzing the dynamic correlation between various performance indicators, thereby achieving a precise description of the performance state of complex systems. Hybrid evaluation methods that integrate multi-source information are among the current mainstream evaluation methods. Evidential Reasoning (ER) rules, as a typical paradigm of hybrid evaluation methods, construct a distributed confidence structure to uniformly represent test indicators as evidence, and achieve multi-source evidence fusion based on orthogonal composition rules, providing a theoretical framework for revealing the global performance of complex systems.

[0004] However, two major challenges exist in practical engineering applications: First, in complex mission scenarios, systems need to perform high-frequency, non-uniformly spaced continuous tests (such as intermittent conditions during spacecraft attitude adjustment), resulting in monitoring data exhibiting significant non-uniform interval characteristics. Second, due to factors such as test environment disturbances, acquisition equipment failures, and storage anomalies, monitoring data is prone to non-random missing data (such as data loss due to communication interruption during a test mission), and traditional data deletion strategies exacerbate the uncertainty of small-sample evaluations. Especially when missing data contains key features, direct removal will cause performance evaluation results to deviate significantly from the true state, thereby affecting the accuracy of judging the system state and the reliability of maintenance decisions. Summary of the Invention

[0005] To address at least one deficiency or improvement requirement in existing technologies, this invention provides a method and apparatus for evaluating the performance of complex systems that takes into account missing data. This method solves the problem that in existing technologies, when performing high-frequency, non-uniformly spaced continuous tests in complex task scenarios, the monitoring data exhibits significant non-uniform interval characteristics and is prone to non-random missing data. Traditional data deletion strategies exacerbate the uncertainty of small sample evaluations, especially when the missing data contains key features, leading to inaccurate performance evaluation results and affecting the accuracy of judging the system status and the reliability of maintenance decisions.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a method for evaluating the performance of complex systems considering missing data is provided, comprising:

[0007] Calculate the estimated values ​​of test indicators, data completeness, target weights, and reliability based on the observation data of test indicators for complex systems;

[0008] Construct reference values ​​for the evaluation rank space, and combine the estimated values, target weights, and reliability to calculate the output utility of complex systems containing missing data;

[0009] The sensitivity of a complex system is calculated based on output utility and data integrity, and the output utility of the complex system is evaluated when data integrity changes.

[0010] In one possible implementation, calculating the estimated values ​​of the test metrics, data completeness, target weights, and reliability based on observed data of the test metrics of the complex system also includes:

[0011] Define state variables based on the missing test indicators to determine whether the observed data at each time step is valid;

[0012] Calculate the estimated values ​​of the test indicators based on the observation time points, state variables, and observation data;

[0013] Data completeness, target weights, and reliability are defined by state variables, state variables, and total number of observation times.

[0014] In one possible implementation, calculating the estimated value of the test index based on the observation time point, state variables, and observation data also includes:

[0015] Constructing missing interval features This represents the time span between the current missing period and the most recent valid observation, and is calculated as follows:

[0016] ;

[0017] Calculate the estimated value of the test index. for:

[0018] ;

[0019] ;

[0020] in, α Indicates the test metrics, Indicators The k Each observation time point for The observed values ​​of the test indicators at each moment; For defined state variables; These are valid observations; This represents the historical mean of the observed values. This is a dynamic decay coefficient, reflecting the ability of recent observations to provide information about currently missing data; It is an exponential decay constant.

[0021] In one possible implementation, data completeness, target weights, and reliability are defined through state variables, total observation time, and the following additional features are included:

[0022] Define data integrity To characterize test metrics α The proportion of complete data is calculated using the following formula:

[0023] ;

[0024] Let the initial weights be... Then, under the condition of missing data, the evaluation test indicators α Target weight With reliability It can be defined as:

[0025] ;

[0026] ;

[0027] in, T This represents the total number of observation times. This indicates the number of unreliable data points for the indicator; For binary markers (if) ,but ,otherwise This is used to determine whether the observed value is within the confidence threshold range; and These are the mean and variance of the observed data, representing their standard values ​​and fluctuation levels, respectively. Adjust the confidence interval range.

[0028] In one possible implementation, a reference value for the evaluation rating is constructed, and the output utility of the complex system containing missing data is calculated by combining the estimated value, target weights, and reliability. This also includes:

[0029] Calculate the assessment confidence level for the corresponding assessment level based on the estimated value and the reference value for the assessment level;

[0030] Based on the ER rule, the output utility of a complex system containing missing data is calculated according to the assessment confidence, target weight, and reliability.

[0031] In one possible implementation, calculating the assessment confidence level corresponding to the assessment level based on the estimated value and the reference value of the assessment level further includes:

[0032] Assess confidence level The following can be calculated:

[0033] ;

[0034] Among them, the assessment level The reference value is , It is to give The quantified utility value satisfies monotonicity: , Indicates the span between levels.

[0035] In one possible implementation, the sensitivity of the complex system is calculated based on output utility and data integrity, and the output utility of the complex system is evaluated using the sensitivity when data integrity changes. This further includes:

[0036] When data integrity Generate tiny increments The change in output utility is Then the sensitivity function is defined as:

[0037] ;

[0038] It can be represented as:

[0039] ;

[0040] in, To incorporate the change in confidence level, This serves as a reference value for the utility of the fusion confidence level.

[0041] According to a second aspect of the present invention, a complex system performance evaluation apparatus that takes into account missing data is also provided, comprising:

[0042] The parameter calculation module is configured to calculate the estimated values ​​of test indicators, data completeness, target weights, and reliability based on the observation data of test indicators of complex systems.

[0043] The output utility module is configured to construct a reference value for the evaluation rank space and to calculate the output utility of a complex system containing missing data by combining the estimated value, target weights, and reliability.

[0044] The performance evaluation module is configured to calculate the sensitivity of a complex system based on output utility and data integrity, and to evaluate the output utility of the complex system when data integrity changes.

[0045] According to a third aspect of the present invention, a complex system performance evaluation apparatus considering missing data is also provided, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of any of the above-described complex system performance evaluation methods considering missing data.

[0046] According to a fourth aspect of the invention, a storage medium is also provided, which stores a computer program executable by a complex system performance evaluation device that considers missing data. When the computer program is run on the complex system performance evaluation device that considers missing data, the device performs the steps of any of the above-described complex system performance evaluation methods that consider missing data.

[0047] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0048] This invention provides a performance evaluation method for complex systems that considers missing data. By introducing comprehensive indicators such as data completeness, target weights, and reliability, it characterizes the impact of missing data from multiple dimensions. Instead of simply removing or crudely filling in missing data, it models and analyzes the missing data itself as information, effectively alleviating the limitations of traditional methods when facing non-random missing data. Even with small sample sizes or severe data loss, it can still provide highly reliable performance evaluation results by making reasonable use of existing information, improving the method's adaptability in complex task environments. A comprehensive evaluation framework integrating estimated values, data completeness, target weights, and reliability is constructed. This framework not only captures more comprehensive multi-faceted characteristics of system performance but also achieves more robust and balanced performance evaluation through dynamic weight adjustment and reliability verification even with incomplete data. This ensures that the evaluation results maintain high interpretability and stability even when the degree of data loss fluctuates, avoiding overall misjudgment due to the absence or bias of a single indicator. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating an embodiment of the performance evaluation method for complex systems considering missing data provided by the present invention;

[0051] Figure 2 Provided by the present invention Figure 1 A schematic flowchart of an embodiment of step S101;

[0052] Figure 3 A schematic diagram illustrating variations of an embodiment of the estimated data for the EST evaluation index provided by the present invention;

[0053] Figure 4 A schematic diagram of the confidence distribution of the EST evaluation index provided by the present invention;

[0054] Figure 5 A schematic diagram of the confidence distribution of the EST system provided by the present invention;

[0055] Figure 6 A schematic diagram of the output utility of an embodiment of the EST system provided by the present invention;

[0056] Figure 7 A schematic diagram of curves representing an embodiment of the output results with different data integrity at time 10 provided by the present invention;

[0057] Figure 8 A schematic diagram of a curve representing an embodiment of the output results with different data integrity at time 65 provided by the present invention;

[0058] Figure 9 A schematic diagram of an embodiment of the performance evaluation device for complex systems considering missing data provided by the present invention;

[0059] Figure 10 A schematic diagram of the structure of a complex system performance evaluation device that takes into account missing data, provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0061] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0062] This invention provides a method and apparatus for evaluating the performance of complex systems that take into account missing data, which will be described below.

[0063] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the performance evaluation method for complex systems considering missing data provided by the present invention. In a specific embodiment of the present invention, a performance evaluation method for complex systems considering missing data is disclosed, including:

[0064] S101. Calculate the estimated values ​​of the test indicators, data completeness, target weights, and reliability based on the observation data of the test indicators of the complex system.

[0065] S102. Construct reference values ​​for the evaluation gradation space, and combine the estimated values, target weights, and reliability to calculate the output utility of the complex system when missing data is included.

[0066] S103. Calculate the sensitivity of the complex system based on the output utility and data integrity, and use the sensitivity to evaluate the output utility of the complex system when the data integrity changes.

[0067] In the above embodiments, the estimated values ​​of missing test indicators are estimated to restore system performance indicator values ​​that are as close as possible to the true state. This not only preserves the validity of existing data but also compensates for the gaps caused by missing information through model inference. Data completeness measures the coverage of data in the current test cycle or evaluation unit, that is, the proportion of valid data to the total data to be observed, reflecting the severity of data missing.

[0068] Based on the system performance evaluation objectives and the importance of each test indicator, corresponding weights are assigned to each indicator. This can be done using expert experience, historical data analysis, or optimization algorithms to ensure that key performance indicators remain dominant in the overall evaluation even when some are missing. The degree of consistency between the current estimate and the actual value, i.e., the credibility level of the evaluation results, is measured. By analyzing data distribution characteristics, model prediction errors, and historical validation results, the reliability of each indicator or the overall evaluation is calculated, providing quality assurance for the final output.

[0069] A reference value system for the evaluation level space is constructed to classify the performance of complex systems into multiple levels (such as excellent, good, average, poor, etc.), providing an intuitive classification standard for performance evaluation. Then, combining the previously calculated estimates, target weights, and reliability, a comprehensive output utility index for system performance under conditions including missing data is calculated. This index reflects the current performance level of the system, integrates multi-dimensional information such as data quality and index importance, and forms a global and comprehensive performance evaluation result.

[0070] To further enhance the adaptability and stability of the evaluation method under dynamically changing data environments, a sensitivity analysis mechanism is introduced. Specifically, based on the calculated output utility and data completeness, the sensitivity of data completeness changes to output utility is further analyzed, i.e., the sensitivity index of the system is calculated. This index reflects the potential impact on system performance evaluation results when data missingness fluctuates. In practical applications, when data completeness changes due to variations in test conditions, system state fluctuations, or external interference, the pre-calculated sensitivity index can be used to quickly assess and predict trends in output utility, enabling dynamic monitoring and early warning of system performance.

[0071] Compared with existing technologies, this embodiment provides a complex system performance evaluation method that considers missing data. By introducing comprehensive indicators such as data completeness, target weights, and reliability, it characterizes the impact of missing data from multiple dimensions. Instead of simply removing or coarsely filling in missing data, it models and analyzes the missing data itself as information, effectively alleviating the limitations of traditional methods when facing non-random missing data. Even with small sample sizes or severe data loss, it can still provide highly reliable performance evaluation results by making reasonable use of existing information, improving the method's adaptability in complex task environments. A comprehensive evaluation framework integrating estimated values, data completeness, target weights, and reliability is constructed. This framework not only captures more comprehensive multi-faceted characteristics of system performance but also achieves more robust and balanced performance evaluation through dynamic weight adjustment and reliability verification even with incomplete data. This ensures that the evaluation results maintain high interpretability and stability even when the degree of data loss fluctuates, avoiding overall misjudgment due to the absence or bias of a single indicator.

[0072] Please see Figure 2 , Figure 2 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S101. In some embodiments of the present invention, calculating the estimated value of the test index, data completeness, target weight, and reliability based on the observed data of the test index of the complex system further includes:

[0073] S201. Define state variables based on the missing test indicators to determine whether the observed data at each moment is valid;

[0074] S202. Calculate the estimated values ​​of the test indicators based on the observation time points, state variables, and observation data;

[0075] S203. Define data completeness, target weight, and reliability through state variables, state variables, and total number of observation times.

[0076] In the above embodiments, based on the actual observation of the test indicators, it is determined whether there is valid data at each time step (or each sampling point): if there is, the corresponding state variable is marked as "valid"; if the data is missing due to test interruption, equipment failure, communication delay or other reasons, it is marked as "invalid", so as to avoid simply regarding missing data as "zero value" or "default value", thereby preventing information misleading and evaluation bias.

[0077] For moments marked as "valid", actual observation data can be used directly; for moments marked as "invalid", the index value at that moment is inferred using a pre-trained model or statistical method based on the distribution characteristics, temporal relationships, or system operation rules of the surrounding valid data. It is necessary to consider the temporal correlation of the data, the dynamic characteristics of the system, and possible noise interference to ensure that the estimated value is as close as possible to the true state.

[0078] Traditional methods often ignore the specific distribution of missing data when calculating similar parameters, leading to overly coarse or static assessments of data completeness, weight allocation, and reliability. This method, however, introduces state variables to achieve a refined description and dynamic tracking of missing data, making the calculations of data completeness, target weights, and reliability more closely reflect real-world data environments. This improves the accuracy of the assessment results and enhances the method's adaptability to complex data environments, particularly maintaining the stability and reliability of the assessment results even when the degree of missing data fluctuates significantly.

[0079] In some embodiments of the present invention, calculating the estimated value of the test index based on the observation time point, state variables, and observation data further includes:

[0080] Constructing missing interval features This represents the time span between the current missing period and the most recent valid observation, and is calculated as follows:

[0081] (1)

[0082] Calculate the estimated value of the test index. for:

[0083] (2)

[0084] (3)

[0085] in, α Indicates the test metrics, Indicators The k Each observation time point for The observed values ​​of the test indicators at each moment; For defined state variables; These are valid observations; This represents the historical mean of the observed values. This is a dynamic decay coefficient, reflecting the ability of recent observations to provide information about currently missing data; It is an exponential decay constant.

[0086] In the above embodiments, state variables are defined. express exist Does an observation exist at time:

[0087] (4)

[0088] In some embodiments of the present invention, data completeness, target weight, and reliability are defined by state variables, state variables, and total number of observation times, and the invention further includes:

[0089] Define data integrity To characterize test metrics α The proportion of complete data is calculated using the following formula:

[0090] (5)

[0091] Let the initial weights be... Then, under the condition of missing data, the evaluation test indicators α Target weight With reliability It can be defined as:

[0092] (6)

[0093] (7)

[0094] in, T This represents the total number of observation times. This indicates the number of unreliable data points for the indicator; For binary markers (if) ,but ,otherwise This is used to determine whether the observed value is within the confidence threshold range; and These are the mean and variance of the observed data, representing their standard values ​​and fluctuation levels, respectively. Adjust the confidence interval range.

[0095] In some embodiments of the present invention, constructing reference values ​​for evaluation levels and calculating the output utility of a complex system containing missing data by combining estimated values, target weights, and reliability, further includes:

[0096] Calculate the assessment confidence level for the corresponding assessment level based on the estimated value and the reference value for the assessment level;

[0097] Based on the ER rule, the output utility of a complex system containing missing data is calculated according to the assessment confidence, target weight, and reliability.

[0098] In the above embodiments, the evaluation level space is first constructed: (For example, the health status of equipment can be defined as...) ), assessment level The reference value is .

[0099] To reflect the credibility of the assessment results (i.e., the reliability of the assessment results under the current data quality and missing data conditions), the method further calculates an assessment confidence index. This index not only depends on which grade range the estimated value falls into, but also considers factors such as data completeness, estimation error range, reliability, and the distribution characteristics of the current missing data. A higher confidence index indicates a more reliable grade judgment under the current missing data conditions; conversely, a lower confidence index indicates greater uncertainty in the assessment results, requiring careful interpretation or further verification with other information.

[0100] Evidence Reasoning (ER) rules, also known as ER rules, are a mathematical method widely used in the fields of multi-source information fusion, uncertainty reasoning, and decision support. Its core idea is to obtain a global and comprehensive evaluation result by weighted fusion of different evidence (in this case, the confidence and weight information of each evaluation indicator) and combining the reliability of each piece of evidence (i.e., reliability) while considering uncertainty and information conflict.

[0101] By using the ER rule to weight and fuse these multidimensional information, an output utility value that can comprehensively reflect the overall performance of the system is finally calculated. This value not only includes the evaluation results of each indicator, but also incorporates factors such as data quality, indicator importance, and estimation reliability.

[0102] In some embodiments of the present invention, calculating the assessment confidence level corresponding to the assessment level based on the estimated value and the reference value of the assessment level further includes:

[0103] Assess confidence level The following can be calculated:

[0104] (8)

[0105] Among them, the assessment level The reference value is , It is to give The quantified utility value satisfies monotonicity: , Indicates the span between levels.

[0106] In the above embodiment, it is assumed that the system has two test metrics. and After introducing missing data, based on ER rules, and Fusion confidence The calculation is as follows:

[0107] (9)

[0108] in, The test is marked as .

[0109] Assumption The utility reference value is The output utility of complex systems containing missing data for:

[0110] (10)

[0111] In some embodiments of the present invention, the sensitivity of a complex system is calculated based on output utility and data integrity, and the output utility of the complex system is evaluated using the sensitivity when data integrity changes. The method further includes:

[0112] When data integrity Generate tiny increments The change in output utility is Then the sensitivity function is defined as:

[0113] (11)

[0114] It can be represented as:

[0115] (12)

[0116] in, To incorporate the change in confidence level, This serves as a reference value for the utility of the fusion confidence level.

[0117] In the above embodiments, the incremental difference of the fusion confidence is... To expand further, The molecular formula is The denominator is The increment difference is:

[0118] (13)

[0119] in, .

[0120] Furthermore, we have:

[0121] (14)

[0122] (15)

[0123] (16)

[0124] in, , .

[0125] In the performance evaluation of complex systems, sensitivity analysis based on ER rules is used to output utility. This provides a quantitative basis for reliability assessment. By defining a sensitivity function... (Equation 11) This method dynamically quantifies data integrity. The impact of minute changes on output utility. Its evaluation mechanism is reflected in three aspects: first, key indicator identification. This involves comparing the data completeness of different indicators (such as...). and Sensitivity value Positioning the effect on output utility The most significant indicators guide resource prioritization; secondly, dynamic weight optimization. Based on... With weight derivative Adaptive adjustment of evidence fusion weights First, suppress interference from low-quality indicators; second, set operation and maintenance thresholds. Through analysis... Follow By leveraging the extreme value characteristics of data integrity, a safe boundary for data integrity is determined, preventing a precipitous drop in output utility. This analysis transforms the physical uncertainties (missing data, noise) at the data layer into risk indicators at the decision-making layer, supporting real-time optimization of maintenance strategies throughout the entire lifecycle of complex systems and significantly improving the robustness of ER rules in multi-source information fusion.

[0126] In real-world scenarios involving complex systems, system evaluation typically involves multi-dimensional test metrics. Assume that... M A set of independent observation indicators Construct a set of missing data parameters containing the following core elements based on equations (1)-(7): Based on the confidence combination rules of equations (8)-(9), the fusion results of the two indicators are iteratively synthesized with the third indicator to construct a multi-level confidence combination rule and calculate the output utility of the complex system. Finally, a sensitivity function is constructed to quantify the dynamic impact of changes in data integrity on the evaluation results, thereby obtaining the optimal performance evaluation results of the complex system.

[0127] A specific embodiment of the present invention is analyzed using an electric servo mechanism as a complex system, as detailed below:

[0128] The electric servo mechanism (ESM), as the core execution terminal of a mechatronic system, integrates a servo control subsystem, a power drive unit, and a mechanical transmission module. Its dynamic response characteristics and steady-state accuracy directly determine the reliability of the equipment system's mission execution. To verify the engineering applicability of the performance evaluation method for complex systems considering missing data proposed in this paper, this section constructs an electric servo testbed (EST). This platform, while retaining the key electromechanical characteristics of actual ESMs, achieves optimized controllability of experimental conditions through structural simplification and modular design.

[0129] Based on the working principle of EST, the following evaluation metrics are set to obtain the overall performance status of the system:

[0130] 1) Transmission Ratio (TR): Describes the speed ratio between the input shaft and the output shaft;

[0131] 2) Zero Position Error (ZPE): The deviation between the actual position and the theoretical zero position when EST is zeroed (unit: Z / °).

[0132] 3) Motor Vibration Signal (MVS): Represents the amplitude value of the vibration signal when the EST is working (unit: V / μm).

[0133] 4) Motor Temperature (MT): Represents the operating temperature value of EST (unit: T / ℃).

[0134] Step 1: Missing Data Estimation

[0135] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the variation of an embodiment of the estimated data for the EST evaluation index provided by the present invention. Through multiple consecutive tests, 90 sets of index test data are obtained. Data is randomly removed from four types of evaluation indexes (ZPE, MVS, MT, TR) according to a preset missing pattern (1 / 3 / 5 / 7 data points) to simulate actual scenarios such as sensor failure and communication packet loss. The missing data can be estimated using the above formula. Figure 3 The experimental data shown indicate that ZPE, MVS, and MT exhibit an upward trend, while TR shows a significant downward trend.

[0136] Based on expert experience and the importance of the indicators, the weight values ​​of the four EST evaluation indicators are set as follows: , , and Further calculations yielded the mixed weight values ​​of the evaluation indicators. With indicator reliability .

[0137] Step 2: Establish an EST system performance evaluation model under data missing conditions

[0138] Each evaluation indicator is assigned three reference levels: “Good (H)”, “Medium (M)”, and “Poor (L)”, with corresponding reference values ​​shown in Table 1.

[0139] Table 1. Reference Values ​​for EST Evaluation Indicators

[0140]

[0141] Please see Figure 4 , Figure 4 This is a schematic diagram of the confidence distribution of the EST evaluation index provided by the present invention. Based on the index prediction data and the reference levels and reference values ​​shown in Table 1, the initial confidence distribution of each index can be obtained.

[0142] Please see Figure 5 , Figure 5 Please refer to the schematic diagram of a confidence distribution of the EST system provided by the present invention. Figure 6 , Figure 6The diagram illustrates an embodiment of the output utility of the EST system provided by the present invention. Based on the reasoning process of the ER method, it is assumed that the utility reference values ​​corresponding to the performance states of the EST are as follows: , and .

[0143] from Figure 5 It can be seen that in the first 20 test moments, the performance of EST showed a slow downward trend; after the 20th moment, the rate of performance degradation from "good (H)" to "medium (M)" gradually accelerated; after the 48th moment, the confidence level of the performance status as "poor (L)" began to appear, indicating that some performance indicators gradually became abnormal and required special attention. The evaluation results truly reflected the actual performance status changes of EST.

[0144] Step 3: Sensitivity analysis of parameters

[0145] Please see Figure 7 , Figure 7 Please refer to the schematic diagram of a curve representing an embodiment of the output results with different data integrity at time 10 provided by the present invention. Figure 8 , Figure 8 This is a schematic diagram of a curve representing an embodiment of the output results with different data integrity at time 65 provided by the present invention. To more intuitively reflect the impact of data integrity on the evaluation results, it is based on... Figure 5 and Figure 6 The output results were used to select time points 10 and 65, with the data integrity score adjusted in progressive steps of 0.01. Specifically, 30 sets of tests were conducted from 0.01 to 0.3 to observe the changes in the confidence distribution and utility value curve, and to analyze and evaluate the stability of the results.

[0146] based on Figure 7 and Figure 8 The confidence distribution and output utility curve shown in the EST performance evaluation results indicate that when data completeness is... As the number of data points increases, the output utility of the EST system also increases, and the system performance gradually improves. Furthermore, as data integrity increases... As the value increases, the rate at which output utility increases gradually slows down (tangent slope), and the trend of improving evaluation results gradually decreases.

[0147] like Figure 7 and Figure 8 As shown, the EST system has good data integrity. In scenarios where data integrity gradually increases, the following key characteristics are exhibited: First, as data integrity increases... As the value increases from 0.01 to 0.3, the output utility exhibits a monotonically increasing trend. Secondly, the rate of utility growth (sensitivity function) ) with data completeness Increased significant decay. Figure 7 In the middle, when data integrity hour, When data integrity hour, It dropped to 0.01.

[0148] To better implement the performance evaluation method for complex systems considering missing data in the embodiments of the present invention, based on the corresponding method for performance evaluation of complex systems considering missing data, please refer to [the relevant documentation / reference]. Figure 9 , Figure 9 This is a schematic diagram of an embodiment of the complex system performance evaluation device considering missing data provided by the present invention. The embodiment of the present invention provides a complex system performance evaluation device 900 considering missing data, comprising:

[0149] The parameter calculation module 910 is configured to calculate the estimated values ​​of the test indicators, data completeness, target weights, and reliability based on the observation data of the test indicators of the complex system.

[0150] Output utility module 920 is configured to construct a reference value for the evaluation rank space and to calculate the output utility of a complex system containing missing data by combining the estimated value, target weights and reliability.

[0151] The performance evaluation module 930 is configured to calculate the sensitivity of a complex system based on output utility and data integrity, and to evaluate the output utility of the complex system when data integrity changes.

[0152] It should be noted that the device 900 provided in the above embodiments can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding content in the above method embodiments, and will not be repeated here.

[0153] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a complex system performance evaluation device considering missing data provided in an embodiment of the present invention. Based on the above-described complex system performance evaluation method considering missing data, the present invention also provides a complex system performance evaluation device considering missing data. This device can be a computing device such as a mobile terminal, desktop computer, laptop, handheld computer, or server. The complex system performance evaluation device 1000 considering missing data includes a processor 1010, a memory 1020, and a display 1030. Figure 10 Only a portion of the components of a complex system performance evaluation device that takes into account missing data are shown. However, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0154] In some embodiments, memory 1020 may be an internal storage unit of the complex system performance evaluation device 1000 that considers missing data, such as a hard disk or memory of the complex system performance evaluation device 1000 that considers missing data. In other embodiments, memory 1020 may be an external storage device of the complex system performance evaluation device 1000 that considers missing data, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the complex system performance evaluation device 1000 that considers missing data. Furthermore, memory 1020 may include both internal storage units and external storage devices of the complex system performance evaluation device 1000 that considers missing data. Memory 1020 is used to store application software and various types of data installed on the complex system performance evaluation device 1000 that considers missing data, such as program code installed on the complex system performance evaluation device 1000 that considers missing data. Memory 1020 may also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 1020 stores a complex system performance evaluation program 1040 that takes into account missing data. The complex system performance evaluation program 1040 that takes into account missing data can be executed by the processor 1010 to implement the complex system performance evaluation method that takes into account missing data in the embodiments of this application.

[0155] In some embodiments, processor 1010 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in memory 1020 or process data, such as executing complex system performance evaluation methods that take into account missing data.

[0156] In some embodiments, display 1030 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 1030 is used to display information from the complex system performance evaluation device 1000 that considers missing data, and to display a user interface for visualization. Components 1010-1030 of the complex system performance evaluation device 1000 that considers missing data communicate with each other via a system bus.

[0157] In one embodiment, when the processor 1010 executes the complex system performance evaluation program 1040 in the memory 1020 that considers missing data, the steps in the complex system performance evaluation method considering missing data described above are implemented.

[0158] This embodiment also provides a computer-readable storage medium storing a complex system performance evaluation program that considers missing data. When executed by a processor, the complex system performance evaluation program that considers missing data performs the following steps:

[0159] Calculate the estimated values ​​of test indicators, data completeness, target weights, and reliability based on the observation data of test indicators for complex systems;

[0160] Construct reference values ​​for the evaluation rank space, and combine the estimated values, target weights, and reliability to calculate the output utility of complex systems containing missing data;

[0161] The sensitivity of a complex system is calculated based on output utility and data integrity, and the output utility of the complex system is evaluated when data integrity changes.

[0162] In summary, this invention provides a performance evaluation method for complex systems that considers missing data. By introducing comprehensive indicators such as data completeness, target weights, and reliability, it characterizes the impact of missing data from multiple dimensions. Instead of simply removing or coarsely filling in missing data, it models and analyzes the missing data itself as information, effectively alleviating the limitations of traditional methods when facing non-random missing data. Even with small sample sizes or severe data loss, it can still provide highly reliable performance evaluation results by rationally utilizing existing information, improving the method's adaptability in complex task environments. A comprehensive evaluation framework integrating estimated values, data completeness, target weights, and reliability is constructed. This framework not only captures more comprehensive multi-faceted characteristics of system performance but also achieves more robust and balanced performance evaluation through dynamic weight adjustment and reliability verification even with incomplete data. This ensures that the evaluation results maintain high interpretability and stability even when the degree of data loss fluctuates, avoiding overall misjudgment due to the absence or bias of a single indicator.

[0163] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0164] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0165] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0169] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0170] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0171] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0172] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating the performance of complex systems that consider missing data, characterized in that, include: Calculate the estimated values ​​of test indicators, data completeness, target weights, and reliability based on the observation data of test indicators for complex systems; Construct reference values ​​for the evaluation gradation space, and combine the estimated values, the target weights, and the reliability to calculate the output utility of the complex system when missing data is included; The sensitivity of the complex system is calculated based on the output utility and the data integrity, and the output utility of the complex system is evaluated using the sensitivity when the data integrity changes. The step of calculating the estimated values ​​of test indicators, data completeness, target weights, and reliability based on the observation data of test indicators of complex systems further includes: Define state variables based on the missing test indicators to determine whether the observation data at each moment is valid; The estimated values ​​of the test indicators are calculated based on the observation time points, the state variables, and the observation data; Data completeness, target weight, and reliability are defined using the observed data, the state variables, and the total number of observation times. The step of calculating the estimated value of the test index based on the observation time point, the state variable, and the observation data further includes: Constructing missing interval features This represents the time span between the current missing period and the most recent valid observation, and is calculated as follows: ； Calculate the estimated value of the test index. for: ； ； in, α Indicates the test metrics, Indicators The k Each observation time point for The observed values ​​of the test indicators at each moment; For defined state variables; These are valid observations; This represents the historical mean of the observed values. This is a dynamic decay coefficient, reflecting the ability of recent observations to provide information about currently missing data; It is an exponential decay constant.

2. The performance evaluation method for complex systems considering missing data as described in claim 1, characterized in that, The definition of data completeness, target weight, and reliability through the state variables, the state variables, and the total number of observation times also includes: Define data integrity To characterize test metrics α The proportion of complete data is calculated using the following formula: ； Let the initial weights be... The target weight for evaluating the test metric α under the condition of missing data is... With reliability It can be defined as: ； ； in, T This represents the total number of observation times. This indicates the number of unreliable data points for the indicator; For binary markers, if ,but ,otherwise This is used to determine whether the observed value is within the confidence threshold range; and These are the mean and variance of the observed data, representing their standard values ​​and fluctuation levels, respectively. These are the observed values ​​of the test indicators; Adjust the confidence interval range.

3. The performance evaluation method for complex systems considering missing data as described in claim 1, characterized in that, The method of constructing a reference value for the evaluation rank space, and calculating the output utility of a complex system containing missing data by combining the estimated value, the target weight, and the reliability, further includes: The assessment confidence level for the corresponding assessment level space is calculated based on the estimated value and the reference value for the assessment level space. Based on the ER rule, the output utility of a complex system containing missing data is calculated according to the assessment confidence, the target weight, and the reliability.

4. The performance evaluation method for complex systems considering missing data as described in claim 3, characterized in that, The step of calculating the assessment confidence level for the corresponding assessment level space based on the estimated value and the reference value of the assessment level space further includes: Assess confidence level The following can be calculated: ； in, Assess the level space for the estimated values ​​of the test metrics. The reference value is , It is to give The quantified utility value satisfies monotonicity: , Indicates the span between levels.

5. The performance evaluation method for complex systems considering missing data as described in claim 1, characterized in that, The step of calculating the sensitivity of the complex system based on the output utility and the data integrity, and using the sensitivity to evaluate the output utility of the complex system when the data integrity changes, further includes: When data integrity Generate tiny increments The change in output utility is Then the sensitivity function is defined as: ； It can be represented as: ； in, To incorporate the change in confidence level, This serves as a reference value for the utility of the fusion confidence level.

6. A performance evaluation device for complex systems that considers missing data, characterized in that, include: The parameter calculation module is configured to calculate the estimated values ​​of test indicators, data completeness, target weights, and reliability based on the observation data of test indicators of complex systems. An output utility module is configured to construct a reference value for an evaluation rank space and to calculate the output utility of a complex system containing missing data by combining the estimated value, the target weight, and the reliability. A performance evaluation module is configured to calculate the sensitivity of the complex system based on the output utility and the data integrity, and to evaluate the output utility of the complex system using the sensitivity when the data integrity changes. The step of calculating the estimated values ​​of test indicators, data completeness, target weights, and reliability based on the observation data of test indicators of complex systems further includes: Define state variables based on the missing test indicators to determine whether the observation data at each moment is valid; The estimated values ​​of the test indicators are calculated based on the observation time points, the state variables, and the observation data; Data completeness, target weight, and reliability are defined using the observed data, the state variables, and the total number of observation times. The step of calculating the estimated value of the test index based on the observation time point, the state variable, and the observation data further includes: Constructing missing interval features This represents the time span between the current missing period and the most recent valid observation, and is calculated as follows: ； Calculate the estimated value of the test index. for: ； ； in, α Indicates the test metrics, Indicators The k Each observation time point for The observed values ​​of the test indicators at each moment; For defined state variables; These are valid observations; This represents the historical mean of the observed values. This is a dynamic decay coefficient, reflecting the ability of recent observations to provide information about currently missing data; It is an exponential decay constant.

7. A performance evaluation device for complex systems that takes into account missing data, characterized in that, It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of the complex system performance evaluation method considering missing data as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, It stores a computer program executable by a complex system performance evaluation device that considers missing data. When the computer program is run on the complex system performance evaluation device that considers missing data, the device performs the steps of the complex system performance evaluation method that considers missing data according to any one of claims 1 to 5.

Images