Intelligent model reliability evaluation system based on improved D-S evidence theory

Abstract

According to the intelligent model reliability evaluation system based on the improved D-S evidence theory provided by the invention, the Wasserstein distance is introduced as a quantification tool for conflicts among evidences, and the measurement can accurately reflect essential differences among probability distributions of different evidence sources; the problem that the measurement is inaccurate when the distribution pattern difference of the traditional Euclidean distance or Jousseme distance is large is solved; meanwhile, Dane entropy is introduced to evaluate the information quality and uncertainty degree of the evidence, cognitive uncertainty and random uncertainty are effectively distinguished, and an information theory basis is provided for subsequent evidence weighting; namely, the Wasserstein distance and the Dane entropy are fused in order to quantify the distribution difference between the evidences and the uncertainty of the evidences more finely and improve the robustness, the accuracy and the interpretability of a reliability evaluation result, so that reliable evaluation support can be provided for landing application of a high-risk intelligent system, and the reliability evaluation method has the advantages of being high in reliability and high in reliability. The method is especially suitable for reliability capability evaluation of intelligent models in high-risk scenes such as complex systems and industrial control systems.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence evaluation technology, and in particular relates to an intelligent model reliability evaluation system based on improved DS evidence theory. Background Technology

[0002] With the increasing application of artificial intelligence (AI) technology in high-security fields such as military command, industrial control, and autonomous driving, the comprehensive capability assessment of intelligent models has become a crucial link in ensuring the safe and stable operation of systems. Especially in complex environments characterized by dynamic confrontation, resource constraints, and multi-source interference, intelligent models not only need high functional accuracy but also need to demonstrate good stability in terms of reliability. Therefore, conducting systematic, multi-dimensional, and interpretable reliability assessment research is of great significance for promoting the reliable deployment of AI in high-risk scenarios.

[0003] Reliability evaluation is not only related to the performance of a model, but also directly relates to the security and credibility of the system. Traditional evaluation metrics such as accuracy, precision, and recall are mostly based on the assumption of independent and identically distributed models, which makes it difficult to comprehensively reflect the model's true performance in complex, dynamic, and even adversarial environments. In practical applications, reliability evaluation evidence often comes from multiple heterogeneous information sources, including real-time sensor data, log audit information, formal verification results, and human expert experience. These pieces of evidence often have different degrees of uncertainty and may conflict significantly with each other. Existing evaluation methods are mostly "island-style" evaluations, with robustness, uncertainty, and fairness metrics being independent of each other, lacking a unified mathematical framework for effective integration and comprehensive evaluation, making it difficult to obtain holistic reliability conclusions. At the same time, traditional metrics cannot accurately quantify the performance degradation of a model when faced with distribution shifts, and cannot directly measure the differences between probability distributions. Especially for deep learning models, their output Softmax probabilities are often overly "confident" and cannot truly reflect cognitive inaccuracies, thus leading to distorted reliability judgments. Furthermore, when evaluation evidence obtained from different dimensions conflicts, there is a lack of effective fusion mechanisms to make a robust and reasonable final judgment. Therefore, constructing a methodology that can systematically, multidimensionally, and quantitatively evaluate the reliability of intelligent models has become an urgent need for both academia and industry.

[0004] Dempster-Shafer (DS) evidence theory, as a classic framework for uncertainty reasoning, is well-suited for handling "unknown" and "uncertain" information and has been widely applied in scenarios such as fault diagnosis, target recognition, and decision support. However, its core combination rules can produce counterintuitive results when dealing with highly conflicting evidence. For example, the famous "Zadeh paradox" shows that when evidence is highly conflicting, the original DS combination rules may lead to synthetic conclusions that contradict common sense. Although researchers have proposed various improvement strategies, such as introducing discount factors, modifying combination rules, and integrating other measures, most of these methods only improve the evidence from a single dimension of conflict measurement or information content, failing to simultaneously consider the differences between evidence and the uncertainty of the evidence itself, resulting in insufficient generalization ability in complex application scenarios.

[0005] Therefore, given the complexity and dynamism exhibited by intelligent models in real-world environments, there is an urgent need for a comprehensive evaluation method that can deeply integrate multi-source heterogeneous evidence and possess stronger resilience and higher interpretability. Summary of the Invention

[0006] To systematically address the bottlenecks in existing intelligent model comprehensive evaluation technologies, such as weak multi-source evidence conflict handling capabilities, inaccurate uncertainty modeling, and low reliability of evaluation results, this invention provides an intelligent model reliability evaluation system based on improved DS evidence theory. This system can achieve comprehensive, quantitative, and interpretable evaluation and decision support for the reliability of intelligent models.

[0007] A reliability evaluation system for intelligent models based on improved DS evidence theory includes: The data acquisition and processing module is used to collect multi-source heterogeneous raw data from the intelligent model to be evaluated and its operating environment in real time or offline, and to standardize the multi-source heterogeneous raw data to obtain normalized indicator data. The basic probability allocation generation module is used to determine the corresponding BPA vector for each indicator data. The BPA vector is a vector composed of the probability that the indicator data belongs to each evaluation level. The evidence conflict detection module is used to obtain the Wasserstein distance between the BPA vectors corresponding to any pairwise combination of indicator data, and to obtain the conflict matrix composed of the Wasserstein distances corresponding to each combination of indicator data. The Evidence Uncertainty Assessment Module is used to obtain the Deng entropy value of the BPA vector corresponding to each indicator data. The evidence weighted fusion module is used to determine the credibility of each indicator data based on the conflict matrix, and to determine the information weight of each indicator data based on the Deng entropy value. Then, it determines the comprehensive weight of each indicator data based on the credibility and information weight. Finally, it uses the comprehensive weight and each indicator data to perform a weighted average to obtain the fused indicator data. The DS evidence synthesis module is used to perform pairwise orthogonal synthesis of each fusion index data according to the Dempster combination rule to obtain the final synthesized index data. The results visualization and weakness identification module is used to determine the comprehensive score of each indicator based on the data of each composite indicator, and to screen and select the indicators with the highest conflict value, the highest uncertainty, and the lowest weight.

[0008] Furthermore, the multi-source heterogeneous raw data includes system fault and availability indicators, redundancy and fault tolerance indicators, performance stability indicators, and task completion quality indicators; Among them, system failure and availability metrics include mean time between failures (MTBF), mean time to recovery (MTBF), number and duration of system service interruptions, graceful degradation trigger records, and performance retention level. Redundancy and fault tolerance metrics include the number of primary / standby module switching times and success rate, the number of heartbeat detection timeouts, and the data synchronization latency between redundant replicas. Performance stability metrics include inference latency variance, throughput fluctuation, memory / CPU / GPU resource utilization, cache hit rate, and garbage collection frequency; The quality indicators for task completion include mAP changes in object detection, accuracy and recall in classification tasks, and the frequency of occurrence of anomaly prediction outputs.

[0009] Furthermore, for quantitative indicators in multi-source heterogeneous raw data, Min-Max standardization or Z-Score standardization is used to transform the values ​​of all quantitative indicators to the [0,1] interval or to transform them to values ​​that conform to the standard normal distribution. The dimensionless values ​​in the [0,1] interval or the values ​​that conform to the standard normal distribution are used as the normalized indicator data corresponding to the quantitative indicators. For qualitative indicators in multi-source heterogeneous raw data, the subjective evaluation corresponding to the qualitative indicators is converted into quantitative scores using expert scoring or fuzzy membership functions. Then, Min-Max standardization or Z-Score standardization is used to convert the quantitative scores corresponding to all qualitative indicators to the [0,1] interval or to values ​​conforming to a standard normal distribution. The dimensionless values ​​in the [0,1] interval or the values ​​converted to conform to a standard normal distribution are used as the normalized indicator data corresponding to the qualitative indicators.

[0010] Furthermore, for quantitative indicators in multi-source heterogeneous raw data, a probability distribution fitting method is used to determine the confidence level of each quantitative indicator belonging to each evaluation level. Then, the confidence levels of each quantitative indicator belonging to each evaluation level are used to form the BPA vector corresponding to each quantitative indicator. The evaluation levels include excellent, good, qualified, defective, and unknown. For qualitative indicators in multi-source heterogeneous raw data, the probability of each qualitative indicator belonging to each evaluation level is determined by using fuzzy membership functions or directly by expert scoring. Then, the probability of each qualitative indicator belonging to each evaluation level is used to form the BPA vector corresponding to each qualitative indicator.

[0011] Furthermore, the method for calculating the reliability of any indicator data is as follows:

[0012] in, To determine the reliability of the i-th indicator data, Let be the Wasserstein distance between the BPA vectors corresponding to the i-th and j-th indicator data, and k be the total number of indicator data.

[0013] Furthermore, the method for calculating the information weight of any indicator data is as follows:

[0014] in, The information content weight of the i-th indicator data is... Let the Deng entropy value be the BPA vector of the i-th indicator data. This is a small constant that is set.

[0015] Furthermore, the method for calculating the comprehensive weight of any indicator data is as follows:

[0016] in, The comprehensive weight of the i-th indicator data is... To determine the reliability of the i-th indicator data, Let k be the information weight of the i-th indicator data, and k be the total number of indicator data.

[0017] Furthermore, the final method for obtaining the synthetic index data is as follows: Perform orthogonal synthesis of any two fusion index data using the following formula:

[0018] in, It is a normalization constant. This is the first fusion indicator data. For the second fusion indicator data, for and The synthesized data; Will The next fusion index data will replace the above formula respectively. and Then, orthogonal synthesis is performed again, and this process is repeated until all fusion index data is synthesized to obtain the final synthesized index data.

[0019] Furthermore, the method by which the results visualization and weakness identification module determines the comprehensive score of each indicator based on the data of each composite indicator is as follows: For the composite index data corresponding to each indicator, select the evaluation level corresponding to the maximum confidence level as the comprehensive evaluation level of each indicator, or calculate the expected utility value of each indicator based on the composite index data corresponding to each indicator as the comprehensive score of each indicator.

[0020] Beneficial effects: 1. This invention provides a reliability evaluation system for intelligent models based on an improved DS evidence theory. It introduces Wasserstein distance as a quantitative tool for assessing conflicts between evidence sources. This metric accurately reflects the essential differences in probability distributions between different evidence sources, overcoming the inaccuracy of traditional Euclidean distance or Jousselme distance when the distribution patterns differ significantly. Simultaneously, it introduces Dunn entropy to evaluate the information quality and uncertainty of the evidence itself, effectively distinguishing between cognitive uncertainty and random uncertainty, providing an information theory basis for subsequent evidence weighting. In other words, to more precisely quantify the distributional differences between evidence and the uncertainty of the evidence itself, and to improve the robustness, accuracy, and interpretability of reliability evaluation results, this invention integrates Wasserstein distance and Dunn entropy. This provides reliable evaluation support for the practical application of high-risk intelligent systems, and is particularly suitable for evaluating the reliability capabilities of intelligent models in high-risk scenarios such as complex systems and industrial control systems.

[0021] 2. This invention provides an intelligent model reliability evaluation system based on an improved DS evidence theory. It proposes an evidence weighting mechanism and a modified combination rule based on conflict degree and information content. By constructing a Wasserstein distance matrix between evidence bodies, the credibility weight of each piece of evidence is calculated. Combined with the information content reflected by the Dunn entropy value, the fusion weight of each evidence body is dynamically generated. Finally, a modified evidence set is generated by weighted averaging and synthesized using the classic DS combination rule. This not only retains the advantages of DS theory in dealing with uncertainty problems, but also effectively suppresses the counterintuitive synthesis results brought about by highly conflicting evidence.

[0022] 3. This invention provides an intelligent model reliability evaluation system based on an improved DS evidence theory. By analyzing the weight distribution and conflict sources of each piece of evidence during the fusion process, it can trace back to the weak links in the original indicator layer, providing clear guidance for the targeted optimization of the intelligent model. This not only achieves "knowing what" but also supports a comprehensive evaluation that "knows why," and has the ability to accurately locate system shortcomings. Attached Figure Description

[0023] Figure 1 A schematic diagram of an intelligent model reliability evaluation system based on improved DS evidence theory provided by the present invention; Figure 2 This invention provides a data flow diagram for an intelligent model reliability evaluation system based on improved DS evidence theory. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0025] This invention innovates holistically at two levels: system architecture and system data processing flow. At the system level, a multimodal evaluation system integrating data acquisition, conflict detection, uncertainty quantification, weighted fusion, and result visualization is constructed, realizing closed-loop processing from raw data to comprehensive decision-making. At the system data processing level, an improved DS evidence theory model integrating Wasserstein distance and Deng Entropy is proposed. By accurately measuring the distributional differences between evidence and the intrinsic uncertainty within the evidence, the fusion effect and interpretive capability are significantly improved in high-conflict and high-uncertainty environments.

[0026] Specifically, such as Figure 1 As shown, an intelligent model reliability evaluation system based on improved DS evidence theory includes: 1. Data acquisition and processing module, used to collect multi-source heterogeneous raw data from the intelligent model to be evaluated and its operating environment in real time or offline, and to standardize the multi-source heterogeneous raw data to obtain normalized indicator data. The multi-source heterogeneous raw data includes multiple metrics such as system failure and availability indicators, redundancy and fault tolerance indicators, performance stability indicators, and task completion quality indicators. System failure and availability indicators include mean time between failures (MTBF), mean time to repair (MTBF), number and duration of system service interruptions, graceful degradation trigger records, and performance maintenance levels. Redundancy and fault tolerance indicators include the number and success rate of primary / standby module switching, the number of heartbeat detection timeouts, and the data synchronization latency between redundant replicas. Performance stability indicators include inference latency variance, throughput fluctuation, memory / CPU / GPU resource utilization, cache hit rate, and garbage collection frequency. Task completion quality indicators include mAP changes in object detection, accuracy and recall in classification tasks, and the frequency of abnormal prediction outputs (such as out-of-bounds values ​​and NaN).

[0027] The aforementioned multi-dimensional indicators can be collected in real time through various means such as log systems, monitoring agents, sensor data streams, database audit records, and system-built-in probes. They are then processed through a unified preprocessing pipeline to remove invalid values, align timestamps, imputate missing values, and perform normalization transformation. Some data will be processed to calculate different indicators according to requirements, such as the number of primary / backup module switching times and success rate, inference delay variance, etc., ultimately forming a standardized dataset suitable for probability allocation and evidence fusion.

[0028] It should be noted that the present invention also requires the collection of raw multi-source heterogeneous raw data to perform cleaning, noise reduction, time alignment, processing of missing values, outliers, duplicate records and formatting, etc. The specific process of standardization processing varies depending on the type of multi-source heterogeneous raw data. For quantitative indicators in multi-source heterogeneous raw data, Min-Max standardization or Z-Score standardization is used to transform the values ​​of all quantitative indicators to the [0,1] interval or to transform them to values ​​that conform to the standard normal distribution. The dimensionless values ​​in the [0,1] interval or the values ​​that conform to the standard normal distribution are used as the normalized indicator data corresponding to the quantitative indicators. For qualitative indicators in multi-source heterogeneous raw data, the subjective evaluation corresponding to the qualitative indicators is converted into quantitative scores using expert scoring or fuzzy membership functions. Then, Min-Max standardization or Z-Score standardization is used to convert the quantitative scores corresponding to all qualitative indicators to the [0,1] interval or to values ​​conforming to a standard normal distribution. The dimensionless values ​​in the [0,1] interval or the values ​​converted to conform to a standard normal distribution are used as the normalized indicator data corresponding to the qualitative indicators.

[0029] In other words, this invention uses min-max normalization, Z-score normalization, or normalization based on expert rules to transform various index data into dimensionless values ​​or values ​​conforming to a standard normal distribution within the interval [0,1] for indices with different dimensions and ranges, providing standard input for subsequent probability allocation.

[0030] 2. Basic probability allocation generation module, used to determine the corresponding BPA vector for each indicator data, wherein the BPA vector is a vector composed of the probability of the indicator data belonging to each evaluation level; In other words, the Basic Probability Assignment (BPA) generation module transforms the normalized indicator data into a basic probability assignment function on the recognition framework. The BPA function is a mapping... It satisfies two conditions: (1) (2) .in, It is the power set of the recognition framework Θ, containing all subsets of Θ (i.e., all single-element propositions and compound propositions). The recognition framework Θ is usually defined as a rating level for a certain aspect of the performance of the intelligent model, such as Θ = {Excellent, Good, Satisfactory, Defective, Unknown}.

[0031] Quantitative indicators: Based on historical data or expert knowledge, probability distribution fitting methods are used to establish a probability distribution model (such as normal distribution, beta distribution, etc.) for each indicator, and to calculate the probability that the current indicator value belongs to each level in the identification framework. For example, for the reliability indicator "mean time to failure", it can be fitted to a Weibull distribution or an exponential distribution, and then the confidence level of the indicator belonging to each evaluation level can be calculated.

[0032] Qualitative indicators: The degree of support of the indicator for each level in the framework is determined by using fuzzy membership functions or direct expert scoring. For example, for indicators that are difficult to quantify precisely, such as "human-machine collaboration conflict rate", a fuzzy rule base is designed (e.g., the degree of membership of "excellent" to "low" conflict rate is 0.8), which is then transformed into a probability distribution on the recognition framework.

[0033] Finally, each indicator generates a BPA vector. In practice, to simplify calculations and facilitate interpretation, it can be agreed that the majority of probability mass is assigned to single-element propositions (i.e., , , , ), and will be allocated to compound propositions (such as The mass of a proposition is set to a very small value (e.g., 0.01) to represent "unknown," or no mass is assigned (i.e., set to 0). Therefore, a BPA vector can primarily consist of probability values ​​on these single-element propositions. For example: , , , d, and (When the quality of a compound proposition is not zero) or (When the quality of a compound proposition is 0).

[0034] 3. Evidence conflict detection module, used to obtain the Wasserstein distance between the BPA vectors corresponding to any pairwise combination of indicator data, and obtain the conflict matrix composed of the Wasserstein distances corresponding to each combination of indicator data. It should be noted that, in order to quantify the degree of conflict between different sources of evidence (i.e., BPAs generated by different indicators), this invention treats each BPA as a probability distribution. For all k evidence bodies (BPAs) in the system, the Wasserstein distance between each pair is calculated. The Wasserstein distance measures the minimum "cost" required to "transfer" one probability distribution to another, and can keenly capture the differences between distribution forms.

[0035] In this invention, each BPA is treated as a discrete probability distribution and is determined based on the Hamming distance on the identification framework (i.e., d(Excellent, Good) = 1, d(Excellent, Satisfactory) = 2, and so on). This distance can be solved using optimization methods such as linear programming. The larger the value, the greater the difference in the distribution patterns of the two pieces of evidence in the evaluation, and the more serious the conflict.

[0036] It should be noted that treating BPA directly as a probability distribution and calculating the Wasserstein distance is an effective engineering approximation, designed to capture the overall distributional differences in grading judgments among different pieces of evidence.

[0037] Distance between all evidence pairs A k×k symmetric matrix W is constructed, which quantifies the degree of conflict between each pair of pieces of evidence, where, The larger the value, the greater the disagreement and the more serious the conflict between the two pieces of evidence.

[0038] 4. Evidence uncertainty assessment module, used to obtain the Deng entropy value of the BPA vector corresponding to each indicator data; It should be noted that Dunn entropy is an extension of Shannon entropy within the framework of DS evidence theory, and is particularly suitable for measuring the uncertainty of compound propositions, such as for assessing the uncertainty and information content contained in each body of evidence (BPA) of this invention.

[0039] For each BPA Calculate its Deng entropy value The formula is:

[0040] Here, A is an element (i.e. a proposition) in the power set of the recognition frame Θ, and |A| represents the cardinality of proposition A (the number of elements it contains).

[0041] The magnitude of the entropy value directly reflects the degree of uncertainty of the evidence itself. The larger the entropy value, the more ambiguous the information provided by the evidence, the lower the certainty, and the higher the uncertainty. It also indicates that the effective information it contains is more dispersed, and its weight in the fusion process should be relatively reduced.

[0042] 5. The evidence weighted fusion module is used to determine the credibility of each indicator data based on the conflict matrix, and to determine the information weight of each indicator data based on the Deng entropy value. Then, based on the credibility and information weight, the comprehensive weight of each indicator data is determined. Finally, the comprehensive weight is used to perform a weighted average with each indicator data to obtain the fused indicator data. It should be noted that the evidence weighting and fusion module aims to assign appropriate weights to each piece of evidence based on the degree of conflict between the evidence and its own uncertainty, and then perform a weighted average to generate a set of revised and more consistent new evidence, namely the fusion index data.

[0043] (1) Calculate the credibility of evidence: For evidence Its credibility The credibility is determined by the average degree of conflict with other evidence. The smaller the conflict, the higher the credibility. The formula for calculating the credibility weight based on conflict is as follows:

[0044] (2) Calculate the amount of information in the evidence: For evidence Its information content It is determined by its entropy. The smaller the entropy, the higher the certainty, the higher the information quality, and the greater the information quantity. The formula for calculating the information quantity weight based on uncertainty is as follows:

[0045] To avoid division by zero errors, a small constant ε is added to the formula, which is usually set according to the magnitude of Deng entropy. In one embodiment of the present invention, the value is 1e-5. (3) Calculate the normalized fusion weights: Combining credibility and information content, for each piece of evidence, multiply its credibility by its information content and normalize it to obtain the final weights:

[0046]

[0047] Among them, the normalized weights Reflects evidence The proportion that should be played in the integration process The weights before normalization.

[0048] Finally, use the calculated weights A weighted average of the original BPA set yields a new set of preprocessed BPAs. Among them, a new set of pre-processed evidence The calculation method is as follows:

[0049] To ensure A valid BPA needs to be normalized:

[0050] This normalized This refers to the fusion index data used in subsequent DS synthesis.

[0051] The weighted fusion process is equivalent to revising the evidence based on its reliability and information content before the formal DS synthesis. This effectively suppresses and weakens the adverse effects of highly conflicting and uncertain evidence, enhances the role of highly credible and certain evidence, and lays a good foundation for subsequent DS synthesis.

[0052] 6. The DS evidence synthesis module is used to perform pairwise orthogonal synthesis of each fusion index data according to the Dempster combination rule to obtain the final synthesized index data. It should be noted that this invention performs orthogonal synthesis on any two fusion index data according to the following formula:

[0053] in, It is a normalization constant used to ignore conflicts between pieces of evidence and to distribute the quality of conflicts proportionally. This is the first fusion indicator data. For the second fusion indicator data, for and The synthesized data; Will The next fusion index data will replace the above formula respectively. and Then, orthogonal synthesis is performed again, and this process is repeated until all fused index data is synthesized, resulting in the final synthesized index data BPA. .

[0054] Therefore, it can be seen that the DS evidence synthesis module receives the weighted BPA and follows the... The rules are fused to generate a comprehensive BPA that reflects the overall reliability level. Because the input evidence has been weighted and optimized, the DS synthesis rules can produce reasonable and intuitive fusion results even when the original evidence is highly conflicting.

[0055] 7. The results visualization and weakness identification module is used to determine the comprehensive score of each indicator based on the data of each composite indicator, and to screen and select the indicators with the highest conflict value, the highest uncertainty, and the lowest weight.

[0056] It should be noted that the results visualization and weakness localization module can present the fusion results in a visual form and reverse analyze the weak links of the localization intelligent model. For example, it can display the final score and comprehensive score of the reliability dimension in the form of radar charts, bar charts, etc., and at the same time show the reliability distribution of each proposition on the recognition framework.

[0057] For tracing highly conflicting evidence, this invention analyzes the conflict matrix W to identify which indicators correspond to the highest conflict values ​​with other evidence. These indicators may be the source of system anomalies or performance bottlenecks.

[0058] For high-uncertainty evidence analysis, this invention examines Deng's entropy. Evidence of abnormally high levels suggests that the corresponding indicators may be inaccurately measured or poorly defined, requiring improvements in measurement methods.

[0059] For low-weighted evidence examination, this invention is assigned an extremely low weight in the weight allocation process. The evidence provided is often based on indicators that either conflict severely with other indicators or are highly uncertain, making them weak points that require close attention and verification.

[0060] Finally, the results visualization and weakness identification module automatically generates a structured evaluation report, which includes a comprehensive score, scores for each dimension, performance level evaluation, and a detailed list of weakness indicators and improvement suggestions.

[0061] As can be seen, the results visualization and shortcoming location module system retrospectively analyzed the intermediate data in the fusion process, checked which evidence bodies showed high conflict values ​​in the conflict matrix, checked which evidence bodies had abnormally high Deng entropy values, and checked which indicators were given low weights in the weighting process. By combining the above modules, reliability evidence from different dimensions and scales can be systematically integrated into a unified and meaningful reliability index, such as... Figure 2 As shown.

[0062] Furthermore, the system of the present invention adopts an embedded edge computing platform as its hardware foundation, and its architecture is as follows: 1. Main Control Computing Unit: Utilizing an NVIDIA Jetson AGX Orin module, it features 256 CUDA cores and 8 NVIDIA Deep Learning Accelerator (NVDLA) cores, providing up to 275 TOPS of INT8 computing performance. This unit is responsible for running the object detection intelligent model (YOLOv8s) and the core algorithms of this evaluation system.

[0063] 2. Backup Computing Unit: Employs an NVIDIA Jetson TX2 module, interconnected with the main control unit via a PCIe 3.0 x4 link, forming a master-slave heterogeneous redundancy architecture. When the core temperature of the main unit exceeds 85°C or the GPU utilization remains above 95% for 3 consecutive minutes, a watchdog circuit triggers task switching.

[0064] 3. Multi-source sensor kit: Visible light camera: Sony IMX577 sensor, resolution 3840×2160, frame rate 30fps Infrared thermal imager: FLIR Boson 640, resolution 640×512, frame rate 60Hz Millimeter-wave radar: TI AWR1843, detection range 200m, resolution 0.1m. 4. Monitoring sensors: Temperature sensor: MAX31875, accuracy ±0.5℃, for acquiring GPU core temperature and PCB board temperature. Current monitoring chip: INA226, accuracy ±0.5%, monitors power consumption on 12V / 5V / 3.3V power rails. Memory health monitoring: Read DDR4 error correction code count via EDAC interface 5. Communication module: 5G communication: Quectel RM500Q-GL module, supporting SA / NSA dual-mode. Beidou RDSS: UB482 module from Hexin Xingtong, communication frequency 1 time / minute 6. Storage Unit: Samsung PM991a NVMe SSD, 1TB capacity, sequential write speed 1.5GB / s II. Software Platform Construction The software stack adopts a layered architecture: 1. Operating System Layer: Ubuntu 20.04 LTS with PREEMPT_RT real-time kernel patch, kernel version 5.10.104-rt64 2. Runtime environment: CUDA 11.4 + cuDNN 8.2.4 TensorRT 8.5.1.7 OpenCV 4.5.2 with GPU acceleration 3. Application Framework Layer: ROS 2 Humble Hawksbill Python 3.8.10 4. Algorithm Implementation Layer: Deep learning model: Convert YOLOv8s.pt to TensorRT engine Evidence processing library: Integrates POT 0.8.2 for Wasserstein distance calculation The following section uses a certain type of UAV performing a target detection task in a field environment as an example to provide a detailed description of the intelligent model reliability evaluation system based on the improved DS evidence theory provided by this invention.

[0065] 1. Data Acquisition (60-second time window): GPU temperature: 72℃ (normalized value 0.68) Inference frame rate: 22.5 FPS (normalized value 0.85) Memory utilization: 78% (normalized value 0.62) Redundancy handover success rate: 100% (normalized value 1.0) Communication error rate: 1.2e-5 (normalized value 0.92) 2. BPA generation: Taking 'GPU temperature' as an example, its normalized value is 0.68. We use the trapezoidal membership function to convert it to BPA, and the membership degree of each level is calculated as follows: Low level: Defined interval [0, 0.5], membership degree is 1 when X≤0.5, and membership degree is 0 when X≥0.7. Therefore .

[0066] Medium level: Defined interval [0.3, 0.8], where the membership degree is 0 when X = 0.3 and 1 when X = 0.8. Therefore... .

[0067] High level: Defined interval [0.6, 1], membership degree is 0 when X≤0.6, and membership degree is 1 when X≥1. Therefore .

[0068] Normalizing the membership vector [0.1, 0.76, 0.2] yields BPA: m(Low)=0.1 / (0.1 + 0.76 + 0.2) ≈ 0.094 m(Medium)=0.76 / (0.1 + 0.76 + 0.2) ≈ 0.717 m(High) = 0.2 / (0.1 + 0.76 + 0.2) ≈ 0.189 After rounding, the BPA for the GPU temperature is: {'Low': 0.09, 'Medium': 0.72, 'High': 0.19}. This result is highly consistent with the BPA automatically generated by the system, verifying the effectiveness and consistency of this method.

[0069] The BPA results automatically generated by the system are as follows: GPU temperature: {'High': 0.19, 'Medium': 0.72, 'Low': 0.09} Inference frame rate: {'High': 0.88, 'Medium': 0.11, 'Low': 0.01} Memory usage: {'High': 0.08, 'Medium': 0.75, 'Low': 0.17} Redundancy switching: {'High': 0.95, 'Medium': 0.05, 'Low': 0.00} Communication quality: {'High': 0.82, 'Medium': 0.16, 'Low': 0.02} 3. Fusion process: The Wasserstein distance matrix shows that the maximum conflict (0.42) exists between the evidence of GPU temperature and inference frame rate. Deng's entropy calculation shows that the information quality of the redundant switching evidence is the highest (entropy value 0.21). Weighted evidence weight distribution: [0.18, 0.22, 0.15, 0.28, 0.17] 4. Final Result: BPA after fusion: {'High': 0.76, 'Medium': 0.19, 'Low': 0.05} Reliability rating: High (76% confidence level) Weakness identified: GPU temperature has the lowest weighting (0.18), so optimizing the thermal design is recommended. Therefore, it can be seen that the present invention can still obtain reasonable and stable fusion results and accurately locate system shortcomings even in the presence of conflicting evidence (GPU temperature indication has moderate reliability, while other indicators have high reliability) by quantifying conflicts and uncertainties.

[0070] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A reliability evaluation system for intelligent models based on improved DS evidence theory, characterized in that, include: The data acquisition and processing module is used to collect multi-source heterogeneous raw data from the intelligent model to be evaluated and its operating environment in real time or offline, and to standardize the multi-source heterogeneous raw data to obtain normalized indicator data. The basic probability allocation generation module is used to determine the corresponding BPA vector for each indicator data. The BPA vector is a vector composed of the probability that the indicator data belongs to each evaluation level. The evidence conflict detection module is used to obtain the Wasserstein distance between the BPA vectors corresponding to any pairwise combination of indicator data, and to obtain the conflict matrix composed of the Wasserstein distances corresponding to each combination of indicator data. The Evidence Uncertainty Assessment Module is used to obtain the Deng entropy value of the BPA vector corresponding to each indicator data. The evidence weighted fusion module is used to determine the credibility of each indicator data based on the conflict matrix, and to determine the information weight of each indicator data based on the Deng entropy value. Then, it determines the comprehensive weight of each indicator data based on the credibility and information weight. Finally, it uses the comprehensive weight and each indicator data to perform a weighted average to obtain the fused indicator data. The DS evidence synthesis module is used to perform pairwise orthogonal synthesis of each fusion index data according to the Dempster combination rule to obtain the final synthesized index data. The results visualization and weakness identification module is used to determine the comprehensive score of each indicator based on the data of each composite indicator, and to screen and select the indicators with the highest conflict value, the highest uncertainty, and the lowest weight.

2. The intelligent model reliability evaluation system based on improved DS evidence theory as described in claim 1, characterized in that, The multi-source heterogeneous raw data includes system fault and availability indicators, redundancy and fault tolerance indicators, performance stability indicators, and task completion quality indicators. Among them, system failure and availability metrics include mean time between failures (MTBF), mean time to recovery (MTBF), number and duration of system service interruptions, graceful degradation trigger records, and performance retention level. Redundancy and fault tolerance metrics include the number of primary / standby module switching times and success rate, the number of heartbeat detection timeouts, and the data synchronization latency between redundant replicas. Performance stability metrics include inference latency variance, throughput fluctuation, memory / CPU / GPU resource utilization, cache hit rate, and garbage collection frequency; The quality indicators for task completion include mAP changes in object detection, accuracy and recall in classification tasks, and the frequency of occurrence of anomaly prediction outputs.

3. The intelligent model reliability evaluation system based on improved DS evidence theory as described in claim 1, characterized in that, For quantitative indicators in multi-source heterogeneous raw data, Min-Max standardization or Z-Score standardization is used to transform the values ​​of all quantitative indicators to the [0,1] interval or to transform them to values ​​that conform to the standard normal distribution. The dimensionless values ​​in the [0,1] interval or the values ​​that conform to the standard normal distribution are used as the normalized indicator data corresponding to the quantitative indicators. For qualitative indicators in multi-source heterogeneous raw data, the subjective evaluation corresponding to the qualitative indicators is converted into quantitative scores using expert scoring or fuzzy membership functions. Then, Min-Max standardization or Z-Score standardization is used to convert the quantitative scores corresponding to all qualitative indicators to the [0,1] interval or to values ​​conforming to a standard normal distribution. The dimensionless values ​​in the [0,1] interval or the values ​​converted to conform to a standard normal distribution are used as the normalized indicator data corresponding to the qualitative indicators.

4. The intelligent model reliability evaluation system based on improved DS evidence theory as described in claim 1, characterized in that, For quantitative indicators in multi-source heterogeneous raw data, the probability distribution fitting method is used to determine the confidence level of each quantitative indicator belonging to each evaluation level. Then, the confidence levels of each quantitative indicator belonging to each evaluation level are used to form the BPA vector corresponding to each quantitative indicator. The evaluation levels include excellent, good, qualified, defective, and unknown. For qualitative indicators in multi-source heterogeneous raw data, the probability of each qualitative indicator belonging to each evaluation level is determined by using fuzzy membership functions or directly by expert scoring. Then, the probability of each qualitative indicator belonging to each evaluation level is used to form the BPA vector corresponding to each qualitative indicator.

5. The intelligent model reliability evaluation system based on improved DS evidence theory as described in claim 1, characterized in that, The method for calculating the reliability of any indicator data is as follows: in, To determine the reliability of the i-th indicator data, Let be the Wasserstein distance between the BPA vectors corresponding to the i-th and j-th indicator data, and k be the total number of indicator data.

6. The intelligent model reliability evaluation system based on improved DS evidence theory as described in claim 1, characterized in that, The method for calculating the information weight of any indicator data is as follows: in, The information content weight of the i-th indicator data is... Let the Deng entropy value be the BPA vector of the i-th indicator data. This is a small constant that is set.

7. The intelligent model reliability evaluation system based on improved DS evidence theory as described in claim 1, characterized in that, The method for calculating the overall weight of any indicator data is as follows: in, The comprehensive weight of the i-th indicator data is... To determine the reliability of the i-th indicator data, Let k be the information weight of the i-th indicator data, and k be the total number of indicator data.

8. The intelligent model reliability evaluation system based on improved DS evidence theory as described in claim 1, characterized in that, The final method for obtaining the synthetic index data is as follows: Perform orthogonal synthesis of any two fusion index data using the following formula: in, It is a normalization constant. This is the first fusion indicator data. For the second fusion indicator data, for and The synthesized data; Will The next fusion index data will replace the above formula respectively. and Then, orthogonal synthesis is performed again, and this process is repeated until all fusion index data is synthesized to obtain the final synthesized index data.

9. The intelligent model reliability evaluation system based on improved DS evidence theory as described in claim 1, characterized in that, The results visualization and weakness identification module determines the comprehensive score of each indicator based on the data of each composite indicator as follows: For the composite index data corresponding to each indicator, select the evaluation level corresponding to the maximum confidence level as the comprehensive evaluation level of each indicator, or calculate the expected utility value of each indicator based on the composite index data corresponding to each indicator as the comprehensive score of each indicator.

Images