Fault detection and diagnosis general verification evaluation generation method, system and device for mechanical and electronic products

Abstract

The invention discloses a general verification and evaluation generation method, system and device for fault detection and diagnosis of mechanical and electronic products, and relates to the technical field of health diagnosis of electronic products, and the method comprises the steps: determining a first score representing the design capability of a to-be-diagnosed object according to the failure coverage rate and system coverage rate of the to-be-diagnosed object; determining a second score representing the cost capability of the to-be-diagnosed object according to the calculation performance index and the cost-efficiency index of the to-be-diagnosed object; determining a third score representing the algorithm performance of the to-be-diagnosed object according to the fault detection capability index, the fault isolation capability index and the fault diagnosis capability index of the to-be-diagnosed object; and generating a fault detection diagnosis result of the to-be-diagnosed object according to the first score, the second score and the third score. According to the method, the PHM capability of complex equipment such as machinery and electronic products can be systematically diagnosed and evaluated, the diagnosis and evaluation result is objective, accurate and comprehensive, and the operation process is convenient.

Description

Technical Field

[0001] This invention relates to the field of electronic product health diagnosis technology, and in particular to a general verification and evaluation method, system and device for fault detection and diagnosis of mechanical and electronic products. Background Technology

[0002] In engineering applications, mechanical and electronic products are highly complex and comprehensive. Health management evaluation indicators, as a measure of assurance scheme design, play a crucial role in the entire cycle of equipment design, manufacturing, operation, and maintenance. Currently, due to the difficulty in obtaining the failure mechanisms of mechanical and electronic products, the lack of a failure verification environment, and the absence of unified verification methods and performance indicators, the effective verification of research results on health management of electromechanical equipment faces challenges. Summary of the Invention

[0003] The purpose of this invention is to provide a general verification and evaluation method, system, and apparatus for fault detection and diagnosis of mechanical and electronic products, which can solve the above-mentioned problems existing in the prior art.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products, wherein the method includes: Determine the failure coverage and system coverage of the object to be diagnosed; wherein, the object to be diagnosed is a mechanical and electronic product; Based on the failure coverage and the system coverage, a first score characterizing the design capability of the object to be diagnosed is determined; Determine the computational performance and cost-effectiveness indicators of the object to be diagnosed; Based on the aforementioned computational performance and cost-effectiveness indicators, a second score characterizing the cost-effectiveness of the object to be diagnosed is determined; Determine the fault detection capability index, fault isolation capability index, and fault diagnosis capability index of the object to be diagnosed; Based on the fault detection capability index, fault isolation capability index, and fault diagnosis capability index, a third score characterizing the performance of the algorithm for the object to be diagnosed is determined. Based on the first score, the second score, and the third score, a fault detection and diagnosis result for the object to be diagnosed is generated.

[0005] Optionally, the steps of determining the failure coverage and system coverage of the object to be diagnosed include: Determine the number of failures covered by the health status perception of the object to be diagnosed and the total number of failures of the candidate objects for health status perception; The failure coverage rate of the object to be diagnosed is determined based on the ratio of the number of failures to the total number of failures. The system coverage of the object to be diagnosed is determined based on the system function coverage, input condition coverage, execution path coverage, error coverage, performance coverage, and user requirement coverage.

[0006] Optionally, the system function coverage is used to characterize the coverage ratio of all functions of the system of the object to be diagnosed; the condition coverage is used to characterize the coverage ratio of input conditions or parameters; the execution path coverage is used to characterize the proportion of execution paths in the covered system; the error coverage is used to characterize the coverage ratio of expected error conditions and exception handling paths; the performance coverage is used to characterize the performance and load test coverage ratio of the system; and the user requirement coverage is used to characterize the coverage ratio of user requirement points.

[0007] Optionally, the step of determining the computational performance indicators and cost-effectiveness indicators of the object to be diagnosed includes: Based on the computational complexity, CPU processing time, runtime, memory usage, and data processing rate of the object to be diagnosed, the computational performance indicators of the object to be diagnosed are determined; wherein, the computational complexity is used to characterize the time required for the algorithm to run, and the CPU processing time is used to characterize the time spent by the CPU to execute the system software; Calculate the difference between the life cycle cost of the system when unplanned maintenance is managed and the life cycle cost of the system when health status sensing methods are used for management; The cost-effectiveness index of the object to be diagnosed is determined based on the ratio of the difference to the investment in implementing and managing state awareness.

[0008] Optionally, the step of determining the fault detection capability index, fault isolation capability index, and fault diagnosis capability index of the object to be diagnosed includes: Based on the fault detection rate, false alarm rate, missed detection rate, fault detection readiness rate, sensitivity, and stability of the object to be diagnosed, the fault detection capability index of the object to be diagnosed is determined. Based on the fault isolation rate and fault isolation error resolution of the object to be diagnosed, the fault isolation capability index is determined; Based on the diagnostic success function, fault detection threshold metric, confidence metric, confidence stability metric, detection task sensitivity metric, and noise sensitivity metric of the object to be diagnosed, the fault diagnosis capability index is determined.

[0009] This invention provides a general verification and evaluation device for fault detection and diagnosis of mechanical and electronic products, wherein the device includes: The first determining module is used to determine the failure coverage and system coverage of the object to be diagnosed; wherein the object to be diagnosed is a mechanical and electronic product; The second determining module is used to determine a first score characterizing the design capability of the object to be diagnosed based on the failure coverage and the system coverage. The third determining module is used to determine the computational performance indicators and cost-effectiveness indicators of the object to be diagnosed. The fourth determining module is used to determine a second score characterizing the cost-effectiveness of the object to be diagnosed based on the computational performance index and cost-effectiveness index. The fifth determining module is used to determine the fault detection capability index, fault isolation capability index, and fault diagnosis capability index of the object to be diagnosed. The sixth determining module is used to determine a third score characterizing the performance of the algorithm of the object to be diagnosed based on the fault detection capability index, fault isolation capability index and fault diagnosis capability index. The generation module is used to generate the fault detection and diagnosis results of the object to be diagnosed based on the first score, the second score, and the third score.

[0010] Optionally, the first determining module includes: The first submodule is used to determine the number of failures covered by the health status perception of the object to be diagnosed and the total number of failures of the candidate objects for health status perception. The second submodule is used to determine the failure coverage of the object to be diagnosed based on the ratio of the number of failures to the total number of failures. The third submodule is used to determine the system coverage of the object to be diagnosed based on the system function coverage, input condition coverage, execution path coverage, error coverage, performance coverage, and user requirement coverage of the object to be diagnosed.

[0011] Optionally, the system function coverage is used to characterize the coverage ratio of all functions of the system of the object to be diagnosed; the condition coverage is used to characterize the coverage ratio of input conditions or parameters; the execution path coverage is used to characterize the proportion of execution paths in the covered system; the error coverage is used to characterize the coverage ratio of expected error conditions and exception handling paths; the performance coverage is used to characterize the performance and load test coverage ratio of the system; and the user requirement coverage is used to characterize the coverage ratio of user requirement points.

[0012] Optionally, the third determining module includes: The fourth submodule is used to determine the computational performance indicators of the object to be diagnosed based on its computational complexity, CPU processing time, running time, memory, and data processing rate; wherein, the computational complexity is used to characterize the time required for the algorithm to run, and the CPU processing time is used to characterize the time spent by the CPU to execute the system software; The fifth submodule is used to calculate the difference between the life cycle cost of the system when unplanned maintenance is managed and the life cycle cost of the system when health status sensing methods are used for management. The sixth submodule is used to determine the cost-effectiveness index of the object to be diagnosed based on the ratio of the difference to the investment in implementing and managing state awareness.

[0013] Optionally, the fifth determining module includes: The seventh submodule is used to determine the fault detection capability index of the object to be diagnosed based on the fault detection rate, false alarm rate, missed detection rate, fault detection readiness rate, sensitivity and stability of the object to be diagnosed. The eighth submodule is used to determine the fault isolation capability index based on the fault isolation rate and fault isolation error resolution of the object to be diagnosed. The ninth submodule is used to determine the fault diagnosis capability index based on the diagnosis success function, fault detection threshold metric, confidence metric, confidence stability metric, detection task sensitivity metric, and noise sensitivity metric of the object to be diagnosed.

[0014] This invention also provides an electronic device, characterized in that it includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements any of the above-described general verification and evaluation methods for fault detection and diagnosis of mechanical and electronic products.

[0015] This invention also provides a general verification and evaluation generation system for fault detection and diagnosis of mechanical and electronic products, including any of the aforementioned general verification and evaluation generation devices for fault detection and diagnosis of mechanical and electronic products.

[0016] This invention discloses a general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products. The method involves determining the failure coverage and system coverage of the object to be diagnosed; determining a first score characterizing the design capability of the object based on the failure coverage and system coverage; determining a second score characterizing the cost capability of the object based on computational performance and cost-effectiveness indicators; determining fault detection capability indicators, fault isolation capability indicators, and fault diagnosis capability indicators of the object; determining a third score characterizing the algorithm performance of the object based on these indicators; and generating fault detection and diagnosis results based on the first, second, and third scores. This fault detection and diagnosis scheme, based on a three-dimensional evaluation system of design capability, cost capability, and algorithm performance, systematically diagnoses and evaluates the PHM (Prognostics and Health Management) capabilities of complex equipment such as mechanical and electronic products. The diagnostic evaluation results are objective, accurate, and comprehensive, and the operation process is convenient. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of a general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products according to an embodiment of this application. Figure 2 A schematic diagram for generating evaluation indicators for fault detection and diagnosis; Figure 3 This is a flowchart of an adversarial verification method for a fault detection and diagnosis model of mechanical and electronic products, provided in one embodiment of this specification. Figure 4 This is a schematic diagram illustrating the structure of a general verification and evaluation device for fault detection and diagnosis of mechanical and electronic products according to an embodiment of this application. Detailed Implementation

[0018] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0019] This application provides an intelligent health management indicator evaluation system for monitoring and diagnosing performance degradation faults in mechanical and electronic products using integrated engineering data applications; it can also be called a fault detection and diagnosis system. This system can quickly respond to the evaluation needs of PHM (Prognostics and Health Management) for general mechanical and electronic products, providing scientific and accurate health management decisions to improve system performance and reliability.

[0020] Given the complex structure of mechanical and electronic products, the selection of health management indicators presents certain challenges in terms of testing, calculation, and assessment accuracy. Furthermore, obtaining these indicators involves requirements such as setting up the testing environment, time and cost constraints, and appropriate sample size. Therefore, a difficulty and cost analysis of the evaluation indicators proposed in this patent is necessary for different mechanical and electronic products. In addition, due to the complex structure of mechanical and electronic products, which involve multi-system and multi-component health management including hydraulic, fuel, transmission, and electronic systems, an applicability analysis of general verification and evaluation indicators for fault detection and diagnosis is required to form a complete PHM evaluation system for mechanical and electronic products.

[0021] System-level requirements for health management of mechanical and electronic products can generally be categorized into three types: safety, cost, and performance. The goal of a health management indicator evaluation system for mechanical and electronic products is to ensure equipment safety by improving reliability and availability, and to minimize additional costs, thereby maintaining the performance of mechanical and electronic products. Based on the roles of different personnel throughout the entire lifecycle of health status perception, they can be further divided into three categories: 1) Operators (critical component / subsystem / business application, operation management and maintenance personnel, etc.); 2) Decision-makers / supervisors; 3) Engineering designers. Therefore, it is necessary to consider the capability evaluation and measurement indicators for health management of mechanical and electronic products for these different user groups. Based on system design requirements, operational and cost requirements, algorithm performance requirements, and working modes, the health status perception capability evaluation indicators are summarized and categorized as shown in Table 1. Table 1 Evaluation Indicators for Health Management of Mechanical and Electronic Products The following description, in conjunction with the accompanying drawings, details the fault detection and diagnosis method for mechanical and electronic products provided in this application through specific embodiments and application scenarios.

[0022] As attached Figure 1 As shown, the general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products according to an embodiment of this application includes the following steps: Step 101: Determine the failure coverage and system coverage of the object to be diagnosed.

[0023] Among them, the objects to be diagnosed are mechanical and electronic products.

[0024] Failure coverage is the ratio of the number of failures covered by health status awareness to the total number of critical failures that can be candidates for health status awareness. It measures the proportion of software defects or faults that test cases can detect. Specifically, failure coverage refers to the ratio between the number of faults that test cases can trigger during software testing and the total number of actual faults in the software.

[0025] The proportion of critical components / subsystems covered by health status sensing requires dividing the system into several independent modules with high criticality. It is a metric for measuring test comprehensiveness, used to assess whether testing can fully validate all aspects of the system. The method for calculating system coverage may differ depending on whether it's a mechanical or electronic product, but generally involves the aspects shown in the table: Table 2 Increased system coverage generally means more comprehensive testing, enabling more effective discovery of defects in the system. However, improved system coverage does not always equate to improved test quality, as test effectiveness also depends on the quality of test case design, the accuracy of test execution, and the identification and remediation of defects.

[0026] In practical applications, system coverage can serve as an important reference for test planning and execution, but it should be considered in conjunction with other test metrics (such as defect density and test efficiency) to obtain a comprehensive assessment of test quality. Furthermore, system coverage should not be regarded as the sole criterion for test completion, as an excessive pursuit of coverage may lead to neglecting the practicality and effectiveness of test cases.

[0027] Optionally, determining the failure coverage and system coverage of the object to be diagnosed may include the following sub-steps: S1: Determine the number of failures covered by the health status perception of the object to be diagnosed and the total number of failures of the candidate objects for health status perception; S2: Determine the failure coverage of the object to be diagnosed based on the ratio of the number of failures to the total number of failures; S3: Determine the system coverage of the object to be diagnosed based on the system function coverage, input condition coverage, execution path coverage, error coverage, performance coverage, and user requirement coverage.

[0028] Specifically, one feasible approach is to determine the system coverage of the object to be diagnosed by weighted summation of system function coverage, input condition coverage, execution path coverage, error coverage, performance coverage, and user requirement coverage. The weight of each coverage can be flexibly set by those skilled in the art according to actual needs, and no specific restrictions are imposed on this in the embodiments of this application.

[0029] Among them, system function coverage is used to characterize the coverage ratio of all functions of the system to be diagnosed; condition coverage is used to characterize the coverage ratio of input conditions or parameters; execution path coverage is used to characterize the proportion of execution paths in the covered system; error coverage is used to characterize the coverage ratio of expected error conditions and exception handling paths; performance coverage is used to characterize the coverage ratio of system performance and load testing; and user requirement coverage is used to characterize the coverage ratio of user requirement points.

[0030] Step 102: Based on failure coverage and system coverage, determine the first score that characterizes the design capability of the object to be diagnosed.

[0031] In the specific implementation process, the corresponding scores can be determined based on the failure coverage and system coverage, and then the two scores can be weighted and summed to obtain the first score; alternatively, the failure coverage and system coverage can be directly weighted and summed to obtain a coverage value, and the first score can be determined based on the correspondence between the obtained coverage value and the system's preset coverage value range and the score.

[0032] Step 103: Determine the computational performance and cost-effectiveness indicators of the object to be diagnosed.

[0033] An optional method for determining the computational performance and cost-effectiveness metrics of the object to be diagnosed can be as follows: Sub-step 1: Determine the computational performance indicators of the object to be diagnosed based on its computational complexity, CPU processing time, runtime, memory usage, and data processing rate. Computational performance metrics help define the constraints that hardware requirements or software must meet while still satisfying the algorithm's performance requirements, and they also impact the cost of system engineering design and implementation. These metrics primarily include: computational complexity, CPU processing time, runtime, memory usage, and data processing rate.

[0034] Computational complexity is used to characterize the time required for the algorithm to run, which is independent of the software and hardware implementation; CPU processing time is used to characterize the time spent by the CPU executing the system software. CPU processing time is a parameter that measures the time spent by the central processing unit executing the software, and specifies the combined performance of the algorithm / software implementation and its running hardware.

[0035] Sub-step 2: Calculate the difference between the system's life cycle cost when managed using unplanned maintenance and the system's life cycle cost when managed using health status awareness methods; Sub-step 3: Determine the cost-effectiveness index of the object to be diagnosed based on the ratio of the difference to the investment in achieving and managing state perception.

[0036] Cost-effectiveness metrics are used to evaluate the performance of health status sensing from an economic perspective. For example, return on investment (ROI) is calculated as the ratio of the difference between the life cycle cost of the system when managed using unplanned maintenance and the life cycle cost of the system when managed using health status sensing methods, to the investment in implementing and managing health status sensing.

[0037] Step 104: Based on the calculation performance index and cost-effectiveness index, determine the second score that characterizes the cost capability of the object to be diagnosed.

[0038] Step 105: Determine the fault detection capability index, fault isolation capability index, and fault diagnosis capability index of the object to be diagnosed.

[0039] Besides the aforementioned design and cost capabilities, algorithm performance is also a crucial dimension when conducting fault detection and diagnosis for mechanical and electronic products. As a tool for assessing and improving the health of individuals or groups, health management algorithms should possess certain key properties in their evaluation metrics to ensure their effectiveness, reliability, and practicality. The general validation evaluation metrics for fault monitoring and diagnosis of mechanical and electronic products should demonstrate the following capabilities: Figure 2 As shown, the fault detection and diagnosis evaluation indicators can also be called algorithm performance evaluation indicators. For example... Figure 2 As shown, the evaluation indicators for fault detection and diagnosis mainly include: fault detection capability indicators, fault isolation capability indicators, and fault diagnosis capability indicators. Each of these three indicators contains two or more indicators.

[0040] An optional method for determining the fault detection capability index, fault isolation capability index, and fault diagnosis capability index of the object to be diagnosed may be as follows: On the one hand, based on the fault detection rate, false alarm rate, missed detection rate, fault detection readiness rate, sensitivity, and stability of the object to be diagnosed, the fault detection capability indicators of the object to be diagnosed are determined. Secondly, based on the fault isolation rate and fault isolation error resolution of the object to be diagnosed, the fault isolation capability index is determined. Thirdly, based on the diagnostic success function of the object to be diagnosed, the fault detection threshold metric, the confidence metric, the confidence stability metric, the detection task sensitivity metric, and the noise sensitivity metric, the fault diagnosis capability index is determined.

[0041] In a preferred embodiment, the fault detection and diagnosis function of mechanical and electronic products is mainly used to complete early fault detection and fault isolation identification. From the perspective of performance evaluation of diagnostic algorithms, the evaluation methods for fault detection and diagnosis indicators are decomposed into the following three categories: fault detection capability, fault isolation capability, and fault diagnosis capability. Specific indicators are as follows: I. PHM System Detection Capability Indicators: The verification and evaluation of the detection capability indicators of the PHM system serves as a reference for the PHM system's ability to detect faults or abnormal states in electromechanical equipment. According to the content of the verification and evaluation, it can be divided into: verification and evaluation of fault detection accuracy, fault detection delay, fault detection robustness, and fault detection stability.

[0042] The accuracy verification of fault detection specifically includes the following verification parameters: 1) Fault detection rate It refers to the ratio of the number of faults detected by the testing equipment to the total number of faults under specified conditions and within a specified time, expressed as a percentage.

[0043] From a mission safety perspective, for PHM fault diagnosis, in order to achieve autonomous assurance mode, the critical (serious) fault detection rate should reach 100%.

[0044] 2) False alarm rate It refers to the ratio of the number of false alarms occurring under specified conditions and within a specified time to the total number of fault indications within the same time period, expressed as a percentage.

[0045] 3) False negative rate It refers to the ratio of the number of faults that actually occurred but were not detected by the tested equipment under specified conditions and within a specified time to the total number of faults, expressed as a percentage.

[0046] 4) Accuracy ① Mean square error: in, For the true label of the sample, Here is the predicted label for the sample, and n is the number of samples. ② Root mean square error: ③ Mean Absolute Error: in, For the true label of the sample, Let n be the predicted label for the sample, and n be the number of samples.

[0047] ④ Mean absolute percentage error: in, For the true label of the sample, Let n be the predicted label for the sample, and n be the number of samples.

[0048] II. Fault isolation capability indicators of PHM system: Fault isolation refers to the ability to distinguish potentially failing units after a fault is detected. PHM fault isolation typically requires the ability to isolate to one or more LRMs (or LRUs). Fault identification refers to the impact of fault severity on system performance, providing users or relevant personnel with the information required to assess the severity of the fault event and the actions that need to be taken.

[0049] (1) Fault isolation rate Fault isolation rate (FIR) refers to the percentage of all detected faults that a test system can clearly identify, expressed by the following formula: in, The number of detected faults (equivalent to the elements on the diagonal of the ambiguity matrix) that accurately isolate any fault mode to the corresponding component i. This represents the number of detected faults that were not accurately isolated (equivalent to off-diagonal elements).

[0050] (2) Fault isolation error resolution Fault isolation error resolution refers to the ratio of faults that a test system incorrectly identifies out of all detected faults.

[0051] When a fuzzy matrix is ​​normalized, it is equivalent to the sum of probabilities along the off-diagonal lines.

[0052] III. Fault diagnosis capability, evaluated using a fault diagnosis capability validation model, primarily from the following dimensions: (1) Diagnosis success function The fault diagnosis success function is defined as the relationship between the average confidence level of fault diagnosis success and the average fault severity. It can also be called the success function, and can be expressed as: in, This indicates the severity of the fault at time t. This indicates the confidence level of a successful fault diagnosis at this moment. This function can be used to evaluate Boolean confidence levels of 0 or 1, or to evaluate continuous confidence levels within the same interval.

[0053] (2) Fault detection threshold measurement Defined as the ability of a PHM diagnostic system to identify abnormal operating conditions with potential faults at a certain confidence level, such as 95%. Used to calculate the detection threshold metric. The detection threshold is defined as: in, The normalized confidence level represents the severity of the fault, and the success function represents the degree of success. The inverse function of .

[0054] (3) Confidence measurement The overall detection confidence score is defined as the confidence level at which a particular fault mode can be detected across the entire range of fault severity. Based on this metric, a diagnostic system that detects a potential fault with high confidence will receive a higher score, while detecting a fault only when it becomes very severe will result in a lower score. The formula for calculating the overall detection confidence score is: in, The success function at a certain fault depth, i.e. .

[0055] (4) Confidence stability measure The large fluctuations in fault confidence generated by PHM technology are difficult to interpret and therefore undesirable (cannot be used). For example, as the severity of the fault approaches the detection level, diagnostic tools that produce Boolean results indicating whether there is a fault or not will become unstable.

[0056] The confidence stability measure measures the stability of the confidence value throughout the entire fault transition by integrating the difference between each peak. in, and These represent the higher and lower peak confidence levels at each point during the fault transition process, respectively.

[0057] (5) Sensitivity measurement of detection task A PHM system should be able to detect anomalies across the entire range of system operation (tasks), including load and speed. The task sensitivity metric is used to measure the differences in PHM technology output under different task conditions; the formula is as follows: in, and This provides the confidence level output for PHM diagnostic technology under different load or task conditions.

[0058] (6) Noise sensitivity measurement In a given operating environment, sensor data may sometimes be contaminated by noise, potentially interfering with the operation of the fault diagnosis system. The noise sensitivity metric measures the robustness of the PHM fault diagnosis system to noise data and is defined as follows: in, and Let be the success function under noise states 1 and 2, and s be the fault depth. This refers to the false alarm rate.

[0059] Step 106: Based on the fault detection capability index, fault isolation capability index, and fault diagnosis capability index, determine the third score that characterizes the performance of the algorithm of the object to be diagnosed.

[0060] Step 107: Based on the first score, the second score, and the third score, generate the fault detection and diagnosis results for the object to be diagnosed.

[0061] The general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products provided in this application determines the failure coverage and system coverage of the object to be diagnosed; based on the failure coverage and system coverage, a first score characterizing the design capability of the object to be diagnosed is determined; based on computational performance indicators and cost-effectiveness indicators, a second score characterizing the cost capability of the object to be diagnosed is determined; the fault detection capability indicator, fault isolation capability indicator, and fault diagnosis capability indicator of the object to be diagnosed are determined; based on the fault detection capability indicator, fault isolation capability indicator, and fault diagnosis capability indicator, a third score characterizing the algorithm performance of the object to be diagnosed is determined; and based on the first score, second score, and third score, the fault detection and diagnosis result of the object to be diagnosed is generated. This fault detection and diagnosis method, based on an evaluation system of indicators encompassing design capability, cost capability, and algorithm performance, systematically diagnoses and evaluates the PHM capability of complex equipment such as mechanical and electronic products. The diagnostic and evaluation results are objective, accurate, and comprehensive, and the operation process is convenient.

[0062] The following specific example illustrates the general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products provided in this application.

[0063] As a core power component of aircraft, the health status of aero-turbine engines directly impacts flight safety and maintenance costs. Traditional maintenance methods rely on periodic disassembly and inspection, leading to over-maintenance, high failure rate, and high maintenance costs. Existing Predictive Health Management (PHM) systems primarily focus on single-mode fault detection, lacking the ability to collaboratively diagnose multimodal coupled faults, and their validation metrics are scattered, making it difficult to comprehensively evaluate the system's design rationality, economy, and algorithm robustness. Therefore, this invention employs the aforementioned standardized, multi-dimensional PHM system validation method to quantitatively evaluate its comprehensive performance. This invention relates to the field of health management technology for mechanical and electronic products, specifically a validation method for an aero-turbine engine predictive health management (PHM) system based on a three-dimensional evaluation system of design capability, cost capability, and algorithm performance. It is applicable to evaluating the fault detection and diagnosis capabilities of high-reliability equipment. The evaluation is primarily conducted from three dimensions: design capability, cost capability, and algorithm performance.

[0064] I. Design capability verification is as follows: 1) Failure coverage test The simulation engine exhibits five typical faults (blade cracks, coating peeling, bearing wear, fuel injector blockage, and sensor drift). The PHM system covers four of these faults, achieving a coverage rate of 80%. The uncovered faults are multimodal coupled failures (such as concurrent cracking and coating peeling), requiring the expansion of multi-fault collaborative detection logic.

[0065] 2) System coverage test The core subsystems (compressor, combustion chamber, turbine) were tested according to Table 1: Function coverage: 100% (all health monitoring functions activated); Condition coverage: 98% (covering flight altitude 0-12km and rotation speed 50%-110% of rated value); Path coverage: 85% (redundant paths are not fully covered).

[0066] II. Cost Capability Verification (1) Calculation performance indicators Running diagnostic algorithms in an airborne edge computing unit (FPGA+ARM architecture): 1) Single analysis time: 8ms (below the system threshold of 20ms); 2) Peak memory usage: 256MB (in compliance with avionics resource constraints).

[0067] (2) Comparison of cost-effectiveness indicators Taking a single aircraft engine's 5-year cycle as an example: 1) Traditional maintenance: Annual maintenance cost is 1.5 million yuan, and failure loss is 2 million yuan; 2) PHM system: Initial investment of 3 million yuan, annual maintenance cost reduced to 500,000 yuan, and failure loss of 300,000 yuan; 3) Return on Investment (ROI): III. Algorithm Performance Verification (1) Fault detection capability 1) Detection rate: 95% (475 out of 500 simulated faults were detected); 2) False alarm rate: 0.2% (2 false alarms in 1000 hours of normal operation); 3) False negative rate: 5% (concentrated in multimodal coupling faults).

[0068] (2) Fault isolation capability 1) Fault Isolation Rate (FIR): 95.8% (accurately located 455 out of 475 faults); 2) Error resolution: 4.2% (multi-sensor data association logic needs optimization).

[0069] (3) Robustness test 1) Noise sensitivity: After adding 30dB broadband noise, the detection rate drops to 92%, and the false alarm rate rises to 0.6%; 2) Mission sensitivity: Detection rate fluctuates by ±1.5% in different flight phases (takeoff, cruise, landing).

[0070] This specific example can be applied to the engineering verification of a certain type of aircraft turbine engine PHM system. Its evaluation framework can be extended to high-reliability equipment, providing a standardized solution for evaluating the health management capabilities of complex electromechanical products.

[0071] Figure 3 A flowchart is shown for an adversarial verification method for a fault detection and diagnosis model of mechanical and electronic products according to an embodiment of this specification, which specifically includes the following steps.

[0072] Step 302: Receive the first type of verification request and the second type of verification request for mechanical and electronic products.

[0073] Specifically, the first type of verification requirement can be to verify the working condition coverage capability boundary and the fault mode coverage capability boundary of the fault detection model when processing relevant data of mechanical and electronic products through the fault detection model.

[0074] The second type of verification requirement can be to verify the noise robustness, fault degree sensitivity, and sample imbalance sensitivity of the fault detection model when processing relevant data of mechanical and electronic products through the fault detection model.

[0075] Step 304: For the first type of verification requirement, the obtained parameter search domain is divided into at least two subdomains, and the corresponding first verification sample is obtained from the initial sample set according to the number of samples drawn for each subdomain in the current round.

[0076] In an optional implementation, the adversarial verification method for the mechanical and electronic product fault detection and diagnosis model further includes: Based on the accuracy of each verification sample belonging to the target subdomain in the (i-1)th round, determine the number of samples to be extracted for the target subdomain in the current i-th round, where i is a positive integer greater than or equal to 2, and the target subdomain is each of the at least two subdomains.

[0077] Furthermore, determining the number of samples to be extracted from the target subdomain in the current i-th round based on the accuracy of each verification sample belonging to the target subdomain in the (i-1)-th round includes: Based on the accuracy of each validation sample belonging to the target subdomain in the (i-1)th round, determine the first average accuracy corresponding to the target subdomain in the (i-1)th round; Determine the second average accuracy corresponding to the initial sample set; Based on the first average accuracy, the second average accuracy, and the preset basic sampling quantity, the number of samples to be extracted for the target subdomain in the current i-th round is determined.

[0078] Specifically, the parameter search domain can be either the operating condition search domain or the fault mode search domain.

[0079] The operating condition search domain is formed by a combination of a series of operating condition parameters that need to be examined. These operating condition parameters may include temperature stress, humidity stress, vibration stress (maximum vibration, weighted vibration, minimum vibration, continuous cruise vibration, etc.), and working stresses including electrical stress, speed, flow rate, pressure, load, etc.

[0080] The failure mode search domain is formed by a combination of parameters that describe the failure modes that need to be examined.

[0081] By employing a dynamic sample size adjustment method based on the logistic regression function, the more validation samples in the target subdomain cause a significant decrease in the output accuracy of the tested fault detection model, the larger the sample size for that target subdomain will be. Conversely, the more validation samples in the target subdomain cause a significant increase in the output accuracy of the tested fault detection model, the smaller the sample size for that target subdomain will be. This dynamic adjustment method allows for further reduction of the overall number of validation samples while ensuring a sufficient search range. The target subdomain is any one of at least two subdomains.

[0082] Specifically, during the first round of verification, the parameter search domain is either the parameter search domain formed by the initial sample set or a larger search domain that includes the original parameter search domain mentioned above, which can be specified according to the verification needs.

[0083] Once the number of samples to be drawn for each subdomain in the current i-th round is determined, the corresponding first verification sample can be drawn from the initial sample set according to the number of samples drawn.

[0084] The embodiments in this specification can select corresponding verification samples from the fault diagnosis verification sample set collected in actual operation or the fault diagnosis verification sample set generated by simulation model, according to the determined parameter search domain grouping and the sampling number of each group.

[0085] For example, the temperature stress parameter range [0,1000] is divided into 10 subdomains: [0,100), [100,200), ..., [900,1000]. For the subdomain [0,100), if the corresponding sample extraction quantity is 20, it means that 20 first verification samples need to be extracted from the initial sample set. The temperature values ​​contained in these 20 first verification samples all belong to the range [0,100).

[0086] The initial sample set can be the validation set used during the training phase of the fault detection model under test, or other labeled datasets that meet the input requirements of the fault detection model under test and were not used during the training phase. More preferably, a dataset collected in the field after the model is deployed and run can be selected.

[0087] To further reduce computational load, the simulation model can be used to generate the corresponding number of first verification samples after the verification sample requirements are determined.

[0088] Step 306: For the second type of verification requirement, based on the accuracy of each verification sample in the previous round, add at least two types of random noise to each initial sample in the initial sample set to generate the second verification sample for the current round.

[0089] Specifically, an initial sample set can be obtained first to verify the noise robustness of the fault detection model. This initial sample set can be the validation set used during the training phase of the fault detection model, or other labeled datasets that meet the input requirements of the fault detection model but were not used during training. More preferably, a dataset collected in the field after model deployment and operation can be selected.

[0090] In one optional implementation, the step of adding at least two types of random noise to each initial sample in the initial sample set based on the accuracy of each validation sample in the previous round to generate the second validation sample for the current round includes: Based on the accuracy of each verification sample in the (i-1)th round, a first type of random noise is added to the initial samples in the initial sample set to generate a first sub-verification sample, wherein the type of noise contained in each verification sample in the (i-1)th round is the first type. Based on the accuracy of each verification sample in the (i-1)th round, a second type of random noise is added to the initial samples in the initial sample set to generate a second sub-verification sample. The first sub-verification sample and the second sub-verification sample together constitute the second verification sample of the current round.

[0091] Specifically, after obtaining the initial sample set for noise robustness verification of the fault detection model, at least two types of random noise can be added to each initial sample in the initial sample set according to the accuracy of each verification sample in the (i-1)th round to generate the second verification sample for the current i-th round.

[0092] Step 308: Specifically, after determining the first and second verification samples, the second and third verification samples can be input into the fault detection model for processing, and the fault detection model will output the fault detection results of each verification sample in the current round.

[0093] The fault detection result includes at least: the model discrimination result of each validation sample, the predicted probability of the discrimination result, the error with the sample label, and the intermediate process information generated during model inference (e.g., when the fault detection model under test is a deep learning model, the gradient information generated by the model inference).

[0094] In one optional implementation, after obtaining the fault detection result output by the fault detection model, the method further includes: The overall accuracy of each first verification sample in the current i-th round is determined based on the fault detection results. Determine whether the overall accuracy is less than a preset accuracy threshold, or determine whether i is greater than or equal to the first round quantity threshold; If so, proceed to step 310; If the execution result of the step of determining whether the overall accuracy is less than the preset accuracy threshold, or determining whether i is greater than or equal to the first round quantity threshold is negative, then the subdomain extraction ratio corresponding to the parameter search domain is determined. According to the stated subdomain extraction ratio, subdomains are filtered in the at least two subdomains, and the target parameter search domain is determined based on the subdomains obtained through filtering. Increment i by 1, use the target parameter search domain as the parameter search domain, and return to execute the step of dividing the obtained parameter search domain into at least two subdomains.

[0095] Furthermore, determining the subdomain extraction ratio corresponding to the parameter search domain includes: If i is less than the second round quantity threshold, the initial extraction ratio is determined as the subdomain extraction ratio corresponding to the parameter search domain; If i is greater than or equal to the second round quantity threshold, the subdomain extraction ratio corresponding to the parameter search domain is determined based on the overall accuracy of each first verification sample in the current i-th round and the overall accuracy of each verification sample in the (i-1)-th round.

[0096] In one optional implementation, the fault detection results include continuous detection results; Accordingly, determining the overall accuracy of each first verification sample in the current i-th round based on the fault detection results includes: Determine the sample label corresponding to the first target verification sample in the current i-th round, wherein the first target verification sample is each of the first verification samples; Calculate the difference between the fault detection result corresponding to the first target verification sample and the sample label, and determine the first accuracy corresponding to the first target verification sample based on the difference; The overall accuracy corresponding to each first verification sample in the current i-th round is determined based on the first accuracy.

[0097] Alternatively, the fault detection result may include discrete detection results, which may include the probability of each fault category corresponding to the second target verification sample. Accordingly, determining the overall accuracy of each first verification sample in the current i-th round based on the fault detection results includes: The second accuracy corresponding to the second target verification sample is determined by the cross-entropy loss function and in combination with the probability and the number of fault categories, wherein the second target verification sample is each of the first verification samples; The overall accuracy corresponding to each first verification sample in the current i-th round is determined based on the second accuracy.

[0098] Specifically, after obtaining the fault detection results output by the fault detection model, the overall accuracy of each first verification sample in the current i-th round can be determined based on the fault detection results.

[0099] After determining the overall accuracy, it can be determined whether the overall accuracy is lower than the preset accuracy threshold, or whether i has reached the maximum number of iterations T, that is, whether i is greater than or equal to the first round number threshold T, where the preset accuracy threshold and T are specified parameters; if the overall accuracy is lower than the preset accuracy threshold, or i is greater than or equal to the first round number threshold T, then step 310 is executed; if the overall accuracy is greater than or equal to the preset accuracy threshold, and i is less than the first round number threshold T, then subsequent iteration rounds need to be performed.

[0100] In practical applications, the preset accuracy threshold and T can be determined based on the accuracy requirements of the verification and evaluation results and the acceptable computational cost. The smaller the preset accuracy threshold or the larger T is, the more accurate the verification and evaluation results will be, but the greater the computational cost will be.

[0101] In the embodiments of this specification, when the overall accuracy is greater than or equal to the preset accuracy threshold and i is less than the first round quantity threshold T, the subdomain extraction ratio corresponding to the aforementioned parameter search domain can be determined.

[0102] The greater the overall output accuracy of the validation sample set in the previous round decreases, the fewer subdomains will be selected in subsequent rounds. This dynamic adjustment method allows for a further reduction in the total number of validation samples while ensuring a sufficient search range.

[0103] After determining the subdomain extraction ratio corresponding to the aforementioned parameter search domain, subdomain filtering is performed in at least two of the aforementioned subdomains according to this subdomain extraction ratio to extract the subdomains belonging to the subdomains of each validation sample in the current i-th round of validation sample set, and their union is taken as the target parameter search domain to be examined in the next round. Then, this target parameter search domain can be used as the parameter search domain, and the process returns to step 302 to evaluate and validate the fault detection model in the next round.

[0104] In practical applications, if it is necessary to obtain the lower bound verification evaluation results of the operating condition coverage capability or the fault mode coverage capability, then the most difficult subdomain (i.e., the one with the lowest average accuracy) in the current i-th round is extracted, and a certain proportion of subdomains are extracted. If it is necessary to obtain the upper bound verification evaluation results of the operating condition coverage capability or the fault mode coverage capability, then the subdomain with the best output effect (i.e., the one with the highest average accuracy) in the current i-th round is extracted, and a certain proportion of subdomains are extracted.

[0105] In one optional implementation, after generating the corresponding fault detection result, the method further includes: Based on the fault detection results, determine the overall accuracy corresponding to the second verification sample containing the same type of random noise in the current i-th round, wherein the number of overall accuracies is equal to the number of types of random noise, and i is a positive integer greater than or equal to 2; Determine whether the overall accuracy is less than a preset accuracy threshold, or determine whether i is greater than or equal to the first round quantity threshold; If so, proceed to step 310.

[0106] If the execution result of the step of determining whether the overall accuracy is less than the preset accuracy threshold, or determining whether i is greater than or equal to the first round quantity threshold is negative, then the sampling ratio corresponding to the second verification sample containing target type random noise is calculated, wherein the target type is at least one of the at least two types; According to the extraction ratio, the target sample set is obtained by filtering samples from the second verification sample containing random noise of the target type. Increment i by 1, take the target sample set as the initial sample set, and return to execute the step of adding at least two types of random noise to each initial sample in the initial sample set according to the accuracy of each verification sample in the (i-1)th round.

[0107] Furthermore, before calculating the sampling ratio corresponding to the second verification sample containing target-type random noise, the method further includes: Determine whether the overall accuracy of the second verification sample containing random noise of the target type in the current i-th round is less than the preset accuracy threshold; If so, then perform the step of calculating the sampling ratio corresponding to the second verification sample containing target type random noise.

[0108] In one optional implementation, the calculation of the sampling ratio corresponding to the second verification sample containing target-type random noise includes: If i is less than the second round quantity threshold, the initial sampling ratio is determined to be the sampling ratio corresponding to the second verification sample containing random noise of the target type; If i is greater than or equal to the second round quantity threshold, the extraction ratio of the second verification sample containing target type random noise is determined based on the overall accuracy of the second verification sample containing target type random noise in the current i-th round and the overall accuracy of the verification sample containing target type random noise in the (i-1)-th round.

[0109] In one optional implementation, the fault detection results include continuous detection results; Accordingly, determining the overall accuracy of the second verification sample containing the same type of random noise in the current i-th round based on the fault detection result includes: Determine the sample label corresponding to the third target verification sample in the current i-th round, wherein the third target verification sample is each of the second verification samples; Calculate the difference between the fault detection result corresponding to the third target verification sample and the sample label, and determine the first accuracy corresponding to the third target verification sample based on the difference; Based on the first accuracy corresponding to the third target validation sample containing the same type of random noise, determine the overall accuracy corresponding to the third target validation sample containing the same type of random noise in the current i-th round.

[0110] Alternatively, the fault detection result may include discrete detection results, which may include the probability of each fault category corresponding to the fourth target verification sample. Accordingly, determining the overall accuracy of the second verification sample containing the same type of random noise in the current i-th round based on the fault detection result includes: The second accuracy corresponding to the fourth target verification sample is determined by using the cross-entropy loss function and combining the probability and the number of fault categories, wherein the fourth target verification sample is each of the second verification samples; Based on the second accuracy corresponding to the fourth target validation sample containing the same type of random noise, determine the overall accuracy corresponding to the fourth target validation sample containing the same type of random noise in the current i-th round.

[0111] Specifically, after obtaining the fault detection results output by the fault detection model, the overall accuracy corresponding to the second verification sample containing the same type of random noise in the current i-th round can also be determined based on the fault detection results.

[0112] After determining the overall accuracy corresponding to the second validation sample containing various types of random noise, it can be determined whether each overall accuracy is lower than a preset accuracy threshold, or whether i reaches the first round quantity threshold A (the preset accuracy threshold and A are specified parameters). If each overall accuracy is less than the preset accuracy threshold, or i is greater than or equal to the first round quantity threshold A, then step 310 is executed; if there is any overall accuracy greater than or equal to the preset accuracy threshold, and i is less than the first round quantity threshold A, then subsequent iteration rounds are required.

[0113] In practical applications, the preset accuracy threshold and A can be determined based on the accuracy requirements of the verification and evaluation results and the acceptable computational cost. The smaller the preset accuracy threshold or the larger A is, the more accurate the verification and evaluation results will be, but the greater the computational cost will be.

[0114] In the embodiments described in this specification, if any overall accuracy is greater than or equal to a preset accuracy threshold, and i is less than the first round quantity threshold A, the sampling ratio corresponding to the target verification sample containing target type random noise can be calculated. In practical applications, the ratio can be determined according to the following exponential decay function.

[0115] Specifically, before calculating the sampling ratio of the second verification sample containing target type random noise, it is necessary to determine whether the overall accuracy of the second verification sample containing target type random noise in the current i-th round is less than the preset accuracy threshold; if so, the sampling ratio of the second verification sample containing target type random noise is calculated.

[0116] In this way, once it is determined whether the overall accuracy of the second verification sample containing the target type of random noise is less than the preset accuracy threshold, the second verification sample containing the target type of random noise is sampled according to the sampling ratio, and the sampled sample is used as the verification sample for the next round. As for the second verification sample containing other types of random noise, since its corresponding overall accuracy is greater than or equal to the preset accuracy threshold, there is no need to sample it. That is, there is no need to verify the verification sample containing other types of random noise in the next round.

[0117] The greater the decrease in output accuracy in the previous round, the lower the proportion of samples selected. This dynamic adjustment method allows for further reduction of the overall number of validation samples while maintaining the search range.

[0118] After determining the sampling ratio corresponding to the second verification sample containing the target type random noise, samples are screened from the second verification sample containing the target type random noise according to the sampling ratio, so as to extract the sampling ratio of the second verification samples in the current i-th round of verification sample set, and the union of these samples is taken as the target sample set for the next round. Then, the target sample set can be used as the initial sample set, and the process returns to step 302 to evaluate and verify the fault detection model in the next round.

[0119] In practical applications, if it is necessary to obtain the lower bound verification evaluation results of noise robustness, fault sensitivity, and sample imbalance sensitivity, then the most difficult (i.e., the lowest accuracy) sampling proportion of samples in the current i-th round is selected; if it is necessary to obtain the upper bound verification evaluation results of noise robustness, fault sensitivity, and sample imbalance sensitivity, then the best output effect (i.e., the highest accuracy) sampling proportion of samples in the current i-th round is selected.

[0120] Step 310: Based on the fault detection results and preset evaluation indicators, verify and evaluate the fault detection model, and generate corresponding verification and evaluation results.

[0121] Specifically, after obtaining the fault detection results output by the fault detection model, the tested fault detection model can be analyzed and evaluated based on the fault detection results and the evaluation indicators of the fault detection model.

[0122] Based on the fault detection results, evaluation indicators are designed to assess the operational condition coverage capability boundary of the fault detection model. The final round of validation sample set is denoted as the union of the sample sets from all iteration rounds excluding the initial sample set.

[0123] The evaluation index system for the embodiments in this specification is as follows: 1) Fault detection rate, false alarm rate, and missed detection rate under boundary conditions; 2) Single-point fault detection rate under boundary conditions; 3) Fault detection condition sensitivity to countermeasures; 4) Sample variability under boundary conditions.

[0124] Evaluation metrics are used to assess the differences between samples in the final round of validation sample set. When the goal of validation evaluation is to obtain a lower bound on the operating condition coverage capability, i.e., to select difficult samples in the adversarial strategy, a smaller evaluation metric value indicates better operating condition coverage capability of the tested fault detection model. Conversely, when the goal of validation evaluation is to obtain an upper bound on the operating condition coverage capability, i.e., to select good samples in the adversarial strategy, a larger evaluation metric value indicates better operating condition coverage capability of the tested fault detection model.

[0125] The performance of the tested fault detection model at the boundary of operating condition coverage capability is measured by the above-mentioned boundary operating condition fault detection rate (false alarm rate, missed detection rate), boundary operating condition single-point fault detection rate, and boundary fault mode isolation rate (applicable to fault mode coverage capability evaluation). The model is then compared with the corresponding indicators in the initial sample set and model development and design. The smaller the difference, the stronger the model's operating condition coverage capability.

[0126] In addition, boundary analysis can be performed using the aforementioned evaluation metrics. Specifically, the sensitivity of fault detection conditions to adversarial responses and the difference in boundary condition samples can be analyzed. The lower the sensitivity of fault detection conditions to adversarial responses and the higher the difference in boundary condition samples, the stronger the model's condition coverage capability. Furthermore, the boundary samples of condition coverage capability generated in each iteration can be analyzed to identify the model's weaknesses (such as poor performance on a specific type of sample). Based on the relevant analysis results, targeted adjustments can be made to the model's parameters and training process in subsequent iterations.

[0127] Based on the fault detection results, evaluation metrics are designed to assess the fault mode coverage capability boundary of the fault detection model. The final round validation sample set is denoted as the union of the sample sets from all iteration rounds excluding the initial sample set. The evaluation metric system in this embodiment is as follows: 5) Sample variability of boundary fault modes; 6) Isolation rate of boundary fault modes The above steps 5) and 6) evaluate the differences between samples in the final round of validation sample sets. When the goal of validation evaluation is to obtain a lower bound on the failure mode coverage capability, i.e., when difficult samples are selected in the adversarial strategy, the smaller the value of the evaluation index, the better the failure mode coverage capability of the tested fault detection model. Conversely, when the goal of validation evaluation is to obtain an upper bound on the failure mode coverage capability, i.e., when good samples are selected in the adversarial strategy, the larger the value of the evaluation index, the better the failure mode coverage capability of the tested fault detection model.

[0128] Based on the fault detection results, an evaluation index is designed to assess the noise robustness boundary of the fault detection model. The target sample set containing a certain type of noise in the final iteration is denoted as the union of the target sample sets containing type c noise in each iteration, excluding the original sample set. The robustness evaluation index for a certain type of noise in the embodiments of this specification is as follows: 7) Global minimum perturbation radius: The global minimum perturbation radius reflects the maximum perturbation strength that the model input can maintain when it is subjected to type c noise, which is the maximum perturbation strength that can still ensure stable output.

[0129] 8) Local maximum perturbation radius, where the local maximum perturbation radius reflects the maximum perturbation intensity experienced by the model while maintaining input stability across all validation samples. The local maximum perturbation radius takes into account the differences between samples.

[0130] 9) Robust boundary width, where the robust boundary width is the ratio of the difference between the local maximum perturbation radius and the global minimum perturbation radius to the global minimum perturbation radius. This ratio reflects the strength of the influence of individual sample differences on the noise robustness of the tested fault detection model during this noise robustness evaluation. A larger robust boundary width indicates a stronger correlation between the noise robustness of the tested fault detection model and individual sample differences.

[0131] 10) Fault detection noise sensitivity, used to measure the overall robustness of the fault detection model to noise data.

[0132] By analyzing the fault detection results and the aforementioned noise robustness evaluation metrics, the noise robustness boundary of the fault detection model is validated and evaluated. A larger global minimum perturbation radius and local maximum perturbation radius, or a higher fault detection noise sensitivity value, indicates stronger noise robustness of the tested fault detection model. Furthermore, boundary analysis of the fault detection model is performed, specifically examining the robustness boundary width and the fault detection noise sensitivity index. A larger robustness boundary width indicates better representativeness of the validation samples generated in each iteration. Further analysis of the validation samples generated during the iteration process can identify weaknesses in the model (such as poor performance on a specific type of sample). The relevant analysis results can be used for targeted adjustments during subsequent model parameter configuration and training.

[0133] This specification categorizes the verification requirements for noise robustness, operating condition coverage, fault mode coverage, fault severity sensitivity, and sample imbalance sensitivity of mechanical and electronic product fault detection and diagnosis models into two types, and verifies them according to different adversarial iterative verification strategies. For the evaluation requirements of noise robustness, fault severity sensitivity, and sample imbalance sensitivity, a continuously growing adversarial iterative strategy is designed. For the verification requirements of operating condition coverage and fault mode coverage, a discrete grouping adversarial iterative strategy is designed.

[0134] In the continuously increasing adversarial iterative strategy, the interference with the model's normal output increases with each validation round. Based on the accuracy of the fault detection and diagnosis model's output to the validation sample set in the previous round, the validation sample generation parameters and sampling parameters are adversarially adjusted, thereby reducing the model's accuracy in the next prediction round until the model's capability boundary is reached. Specifically, for validation sample generation parameters related to the evaluated capability, such as the noise intensity added when evaluating noise robustness, if the number of iterations is larger or the sample accuracy in the previous round is higher, this parameter will further shift in the direction of increasing the interference with the model. The sampling method and sample size are determined based on the robustness evaluation requirements and the rate of accuracy change during the iteration process.

[0135] In the discrete grouping adversarial iterative strategy, as the validation rounds increase, the search gradually shifts from uniform coverage to samples located at the model's capability boundary. This strategy uniformly divides the search domain of the current round into several groups. Based on the accuracy of the tested model's output and validation requirements of samples in each group, a certain proportion of groups are adversarially selected to form the search domain for the next round. Specifically, if the samples in a certain group cause a greater decrease in the accuracy of the tested model's output, then a larger number of samples will be taken from that group in the next round. The sampling method and sample size in each iteration round are determined based on robustness evaluation requirements and the rate of accuracy change during the iteration process.

[0136] After the iterative verification process is completed, based on the specific requirements for evaluating the capabilities of the fault detection and diagnosis model, a quantitative evaluation index system is established by comprehensively utilizing all historical test information generated during the iteration process to comprehensively assess the boundary capabilities of the tested fault detection and diagnosis model.

[0137] This specification proposes a method for verifying the boundary capabilities of fault detection and diagnosis models to address the issue of different abnormal conditions they may encounter in real-world operating environments. This method has a wide applicability, capable of evaluating various fault detection and diagnosis models, including traditional machine learning models and deep learning models. Based on an adversarial iterative process, this method dynamically adjusts the parameters for constructing validation samples according to the model's actual output, specifically reducing the search space, query count, and computational load. This method fully utilizes the process information from the adversarial iteration to propose a dedicated evaluation index system for the boundary capabilities of fault detection and diagnosis models. Compared to traditional general robustness evaluation indices, this system is more suitable for evaluating the operational condition coverage capability of fault detection and diagnosis models, resulting in more accurate evaluation results.

[0138] This embodiment of the specification receives first and second types of verification requests for mechanical and electronic products. For the first type of request, the parameter search domain is divided into at least two subdomains, and first verification samples are drawn from the initial sample set according to the number of samples drawn from each subdomain in the current round. For the second type of request, random noise is added to each initial sample in the initial sample set based on the accuracy of each verification sample in the previous round to generate the second verification sample for the current round. The first and second verification samples are processed by a fault detection model to generate corresponding fault detection results. Based on the fault detection results and preset evaluation indicators, the fault detection model is verified and evaluated, and corresponding verification and evaluation results are generated. This embodiment of the specification generates different verification samples for different types of verification requests and designs evaluation indicators for evaluating different capabilities of the model. Based on these evaluation indicators and verification samples, the model's multiple capabilities are verified and evaluated, which helps improve the accuracy of the verification and evaluation results, thereby improving the model's actual performance.

[0139] Figure 4 The structural block diagram of the general verification and evaluation device for fault detection and diagnosis of mechanical and electronic products in this application embodiment is shown.

[0140] The general verification and evaluation device for fault detection and diagnosis of mechanical and electronic products according to this application includes the following functional modules: The first determining module 401 is used to determine the failure coverage and system coverage of the object to be diagnosed; wherein the object to be diagnosed is a mechanical and electronic product. The second determining module 402 is used to determine a first score characterizing the design capability of the object to be diagnosed based on the failure coverage and the system coverage. The third determining module 403 is used to determine the computational performance index and cost-effectiveness index of the object to be diagnosed. The fourth determining module 404 is used to determine a second score characterizing the cost-effectiveness of the object to be diagnosed based on the computational performance index and cost-effectiveness index. The fifth determining module 405 is used to determine the fault detection capability index, fault isolation capability index and fault diagnosis capability index of the object to be diagnosed. The sixth determining module 406 is used to determine a third score characterizing the performance of the algorithm of the object to be diagnosed based on the fault detection capability index, the fault isolation capability index and the fault diagnosis capability index. The generation module 407 is used to generate the fault detection and diagnosis result of the object to be diagnosed based on the first score, the second score and the third score.

[0141] Optionally, the first determining module includes: The first submodule is used to determine the number of failures covered by the health status perception of the object to be diagnosed and the total number of failures of the candidate objects for health status perception. The second submodule is used to determine the failure coverage of the object to be diagnosed based on the ratio of the number of failures to the total number of failures. The third submodule is used to determine the system coverage of the object to be diagnosed based on the system function coverage, input condition coverage, execution path coverage, error coverage, performance coverage, and user requirement coverage of the object to be diagnosed.

[0142] Optionally, the system function coverage is used to characterize the coverage ratio of all functions of the system of the object to be diagnosed; the condition coverage is used to characterize the coverage ratio of input conditions or parameters; the execution path coverage is used to characterize the proportion of execution paths in the covered system; the error coverage is used to characterize the coverage ratio of expected error conditions and exception handling paths; the performance coverage is used to characterize the performance and load test coverage ratio of the system; and the user requirement coverage is used to characterize the coverage ratio of user requirement points.

[0143] Optionally, the third determining module includes: The fourth submodule is used to determine the computational performance indicators of the object to be diagnosed based on its computational complexity, CPU processing time, running time, memory, and data processing rate; wherein, the computational complexity is used to characterize the time required for the algorithm to run, and the CPU processing time is used to characterize the time spent by the CPU to execute the system software; The fifth submodule is used to calculate the difference between the life cycle cost of the system when unplanned maintenance is managed and the life cycle cost of the system when health status sensing methods are used for management. The sixth submodule is used to determine the cost-effectiveness index of the object to be diagnosed based on the ratio of the difference to the investment in implementing and managing state awareness.

[0144] Optionally, the fifth determining module includes: The seventh submodule is used to determine the fault detection capability index of the object to be diagnosed based on the fault detection rate, false alarm rate, missed detection rate, fault detection readiness rate, sensitivity and stability of the object to be diagnosed. The eighth submodule is used to determine the fault isolation capability index based on the fault isolation rate and fault isolation error resolution of the object to be diagnosed. The ninth submodule is used to determine the fault diagnosis capability index based on the diagnosis success function, fault detection threshold metric, confidence metric, confidence stability metric, detection task sensitivity metric, and noise sensitivity metric of the object to be diagnosed.

[0145] The general verification and evaluation device for fault detection and diagnosis of mechanical and electronic products provided in this application embodiment systematically diagnoses and evaluates the PHM capabilities of complex equipment such as mechanical and electronic products based on an evaluation system of three dimensions: design capability, cost capability, and algorithm performance. The diagnostic and evaluation results are objective, accurate, and comprehensive, and the operation process is convenient.

[0146] The embodiments provided in this application Figure 4 The general verification and evaluation generator for fault detection and diagnosis of mechanical and electronic products shown can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0147] This invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0148] Memory, used to store computer programs; When the processor executes the program stored in the memory, it implements the general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products as shown in the above method embodiments.

[0149] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0150] The communication interface is used for communication between the aforementioned terminal and other devices.

[0151] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0152] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on an electronic device, cause the electronic device to implement the general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products as described in any of the above embodiments.

[0153] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on an electronic device, enables the electronic device to implement the general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products as described in any of the above embodiments.

[0154] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0155] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products, characterized in that, The method includes: Determine the failure coverage and system coverage of the object to be diagnosed; wherein, the object to be diagnosed is a mechanical and electronic product; Based on the failure coverage and the system coverage, a first score characterizing the design capability of the object to be diagnosed is determined; Determine the computational performance and cost-effectiveness indicators of the object to be diagnosed; Based on the aforementioned computational performance and cost-effectiveness indicators, a second score characterizing the cost-effectiveness of the object to be diagnosed is determined; Determine the fault detection capability index, fault isolation capability index, and fault diagnosis capability index of the object to be diagnosed; Based on the fault detection capability index, fault isolation capability index, and fault diagnosis capability index, a third score characterizing the performance of the algorithm for the object to be diagnosed is determined. Based on the first score, the second score, and the third score, a fault detection and diagnosis result for the object to be diagnosed is generated.

2. The method according to claim 1, characterized in that, The steps to determine the failure coverage and system coverage of the object to be diagnosed include: Determine the number of failures covered by the health status perception of the object to be diagnosed and the total number of failures of the candidate objects for health status perception; The failure coverage rate of the object to be diagnosed is determined based on the ratio of the number of failures to the total number of failures. The system coverage of the object to be diagnosed is determined based on the system function coverage, input condition coverage, execution path coverage, error coverage, performance coverage, and user requirement coverage.

3. The method according to claim 2, characterized in that: The system function coverage is used to characterize the coverage ratio of all functions of the system of the object to be diagnosed; the condition coverage ratio is used to characterize the coverage ratio of input conditions or parameters; the execution path coverage ratio is used to characterize the proportion of execution paths in the covered system; the error coverage ratio is used to characterize the coverage ratio of expected error conditions and exception handling paths; the performance coverage ratio is used to characterize the performance and load test coverage ratio of the system; and the user requirement coverage ratio is used to characterize the coverage ratio of user requirement points.

4. The method according to claim 1, characterized in that, The steps for determining the computational performance indicators and cost-effectiveness indicators of the object to be diagnosed include: Based on the computational complexity, CPU processing time, runtime, memory usage, and data processing rate of the object to be diagnosed, the computational performance indicators of the object to be diagnosed are determined; wherein, the computational complexity is used to characterize the time required for the algorithm to run, and the CPU processing time is used to characterize the time spent by the CPU to execute the system software; Calculate the difference between the life cycle cost of the system when unplanned maintenance is managed and the life cycle cost of the system when health status sensing methods are used for management; The cost-effectiveness index of the object to be diagnosed is determined based on the ratio of the difference to the investment in implementing and managing state awareness.

5. The method according to claim 1, characterized in that, The steps for determining the fault detection capability index, fault isolation capability index, and fault diagnosis capability index of the object to be diagnosed include: Based on the fault detection rate, false alarm rate, missed detection rate, fault detection readiness rate, sensitivity, and stability of the object to be diagnosed, the fault detection capability index of the object to be diagnosed is determined. Based on the fault isolation rate and fault isolation error resolution of the object to be diagnosed, the fault isolation capability index is determined; Based on the diagnostic success function, fault detection threshold metric, confidence metric, confidence stability metric, detection task sensitivity metric, and noise sensitivity metric of the object to be diagnosed, the fault diagnosis capability index is determined.

6. A universal verification and evaluation device for fault detection and diagnosis of mechanical and electronic products, characterized in that, The device includes: The first determining module is used to determine the failure coverage and system coverage of the object to be diagnosed; wherein the object to be diagnosed is a mechanical and electronic product; The second determining module is used to determine a first score characterizing the design capability of the object to be diagnosed based on the failure coverage and the system coverage. The third determining module is used to determine the computational performance indicators and cost-effectiveness indicators of the object to be diagnosed. The fourth determining module is used to determine a second score characterizing the cost-effectiveness of the object to be diagnosed based on the computational performance index and cost-effectiveness index. The fifth determining module is used to determine the fault detection capability index, fault isolation capability index, and fault diagnosis capability index of the object to be diagnosed. The sixth determining module is used to determine a third score characterizing the performance of the algorithm of the object to be diagnosed based on the fault detection capability index, fault isolation capability index and fault diagnosis capability index. The generation module is used to generate the fault detection and diagnosis results of the object to be diagnosed based on the first score, the second score, and the third score.

7. The apparatus according to claim 1, characterized in that, The first determining module includes: The first submodule is used to determine the number of failures covered by the health status perception of the object to be diagnosed and the total number of failures of the candidate objects for health status perception. The second submodule is used to determine the failure coverage of the object to be diagnosed based on the ratio of the number of failures to the total number of failures. The third submodule is used to determine the system coverage of the object to be diagnosed based on the system function coverage, input condition coverage, execution path coverage, error coverage, performance coverage, and user requirement coverage of the object to be diagnosed.

8. The apparatus according to claim 7, characterized in that: The system function coverage is used to characterize the coverage ratio of all functions of the system of the object to be diagnosed; the condition coverage ratio is used to characterize the coverage ratio of input conditions or parameters; the execution path coverage ratio is used to characterize the proportion of execution paths in the covered system; the error coverage ratio is used to characterize the coverage ratio of expected error conditions and exception handling paths; the performance coverage ratio is used to characterize the performance and load test coverage ratio of the system; and the user requirement coverage ratio is used to characterize the coverage ratio of user requirement points.

9. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, communication interface, and memory communicate with each other through a communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the general verification and evaluation method for fault detection and diagnosis of mechanical and electronic products as described in any one of claims 1-5.

10. A general verification and evaluation system for fault detection and diagnosis of mechanical and electronic products, characterized in that, Includes the general verification and evaluation device for fault detection and diagnosis of mechanical and electronic products as described in any one of claims 6-8.

Images