Adversarial verification evaluation method, system and device for fault mode coverage capability boundary of fault detection and diagnosis model

Abstract

The embodiment of the invention provides an adversarial verification evaluation method, system and device for a fault mode coverage capability boundary of a fault detection and diagnosis model. The adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model comprises the steps that an obtained fault mode search domain is divided into at least two sub-domains, and the fault mode search domain is generated by a parameter combination used for describing a target equipment fault mode; determining a sample extraction number corresponding to each sub-domain in the current round, and extracting from the initial sample set according to the sample extraction number to obtain a corresponding target verification sample; processing the target verification sample through a fault detection model to generate a corresponding fault detection result; and based on the fault detection result and a preset evaluation index, verifying and evaluating the fault mode coverage capability of the fault detection model, and generating a corresponding verification and evaluation result.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to an adversarial verification and evaluation method for the boundary of fault mode coverage capability of fault detection and diagnosis models. Background Technology

[0002] Fault detection models can provide early estimates of equipment health and defects based on various information data, enabling timely measures to be taken before catastrophic damage or failure occurs. In addition, fault detection models can also guide the formulation of equipment maintenance plans, thereby extending the lifespan of the equipment.

[0003] Failure mode is a categorized and standardized description of the observable or measurable failure phenomena that occur in a product. It characterizes the external form or morphology of a product when a failure occurs. The failure mode robustness of a fault detection and diagnosis model describes the model's ability to maintain accurate output when encountering any possible failure mode of the object it is detecting and diagnosing.

[0004] Generally, the actual performance of fault detection and diagnosis models degrades when the distribution of sample data changes under certain specific fault modes. For example, because the distribution of signal data collected by sensors differs under different fault modes, the design and training phases of fault diagnosis and detection models cannot cover all fault mode conditions that the model may encounter in actual operation. Therefore, improving the actual performance of fault detection models has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this specification provide an adversarial verification evaluation method for the boundary of fault mode coverage capability of a fault detection and diagnosis model. One or more embodiments of this specification also relate to an adversarial verification evaluation device, a computing device, a computer-readable storage medium, and a computer program for the boundary of fault mode coverage capability of a fault detection and diagnosis model, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, an adversarial verification evaluation method is provided for the fault mode coverage capability boundary of a fault detection and diagnosis model, comprising: The acquired fault mode search domain is divided into at least two subdomains, wherein the fault mode search domain is generated by a combination of parameters used to describe the fault modes of the target device. Determine the number of samples to be extracted for each subdomain in the current round, and extract the corresponding target verification samples from the initial sample set according to the number of samples to be extracted; The target verification sample is processed by a fault detection model to generate corresponding fault detection results; Based on the fault detection results and preset evaluation indicators, the fault mode coverage capability of the fault detection model is verified and evaluated, and corresponding verification and evaluation results are generated.

[0007] Optionally, determining the number of samples to be extracted for each subdomain in the current round includes: Based on the accuracy of each verification sample belonging to the target subdomain in the (i-1)th round, determine the number of samples to be extracted for the target subdomain in the current i-th round, where i is a positive integer greater than or equal to 2, and the target subdomain is each of the at least two subdomains.

[0008] Optionally, determining the number of samples to be extracted from the target subdomain in the current i-th round based on the accuracy of each verification sample belonging to the target subdomain in the (i-1)-th round includes: Based on the accuracy of each validation sample belonging to the target subdomain in the (i-1)th round, determine the first average accuracy corresponding to the target subdomain in the (i-1)th round; Determine the second average accuracy corresponding to the initial sample set; Based on the first average accuracy, the second average accuracy, and the preset basic sampling quantity, the number of samples to be extracted for the target subdomain in the current i-th round is determined.

[0009] Optionally, after generating the corresponding fault detection result, the method further includes: Based on the fault detection results, determine the overall accuracy of each target verification sample in the current i-th round; Determine whether the overall accuracy is less than a preset accuracy threshold; If so, then the step of verifying and evaluating the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators is executed.

[0010] Optionally, after generating the corresponding fault detection result, the method further includes: Determine whether the value of the current round i is greater than or equal to the threshold number of the first round; If so, then the step of verifying and evaluating the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators is executed.

[0011] Optionally, the adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model further includes: If the result of determining whether the overall accuracy is less than the preset accuracy threshold is negative, then the subdomain extraction ratio corresponding to the fault mode search domain is determined. According to the subdomain extraction ratio, subdomains are filtered in the at least two subdomains, and the target fault mode search domain is determined based on the subdomains obtained from the filtering. Increment i by 1, take the target fault mode search domain as the fault mode search domain, and return to execute the step of dividing the acquired fault mode search domain into at least two subdomains.

[0012] Optionally, the adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model further includes: If the result of determining whether the value of the current round i is greater than or equal to the first round quantity threshold is negative, then the subdomain extraction ratio corresponding to the fault mode search domain is determined. According to the subdomain extraction ratio, subdomains are filtered in the at least two subdomains, and the target fault mode search domain is determined based on the subdomains obtained from the filtering. Increment i by 1, take the target fault mode search domain as the fault mode search domain, and return to execute the step of dividing the acquired fault mode search domain into at least two subdomains.

[0013] According to a second aspect of the embodiments of this specification, an adversarial verification and evaluation device for the fault mode coverage capability boundary of a fault detection and diagnosis model is provided, comprising: The partitioning module is configured to divide the acquired fault mode search domain into at least two subdomains, wherein the fault mode search domain is generated by a combination of parameters used to describe the fault modes of the target device. The extraction module is configured to determine the number of samples to be extracted for each subdomain in the current round, and to extract the corresponding target verification samples from the initial sample set according to the number of samples to be extracted. The first generation module is configured to process the target verification sample through a fault detection model to generate a corresponding fault detection result; The second generation module is configured to verify and evaluate the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators, and generate corresponding verification and evaluation results.

[0014] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement any of the steps of the adversarial verification and evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model.

[0015] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of any one of the adversarial verification evaluation methods for the boundary of fault mode coverage capability of a fault detection and diagnosis model.

[0016] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is instructed to perform the steps of the adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model described above.

[0017] This embodiment of the specification divides the acquired fault mode search domain into at least two subdomains, wherein the fault mode search domain is generated by a combination of parameters used to describe the fault modes of the target device; determines the number of samples to be extracted for each subdomain in the current round, and extracts corresponding target verification samples from the initial sample set according to the number of samples extracted; processes the target verification samples through a fault detection model to generate corresponding fault detection results; and verifies and evaluates the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators, generating corresponding verification and evaluation results. By dividing the fault mode search domain into subdomains, determining the number of samples to be extracted for each subdomain, and then extracting samples based on the number of samples extracted, and by designing evaluation indicators for evaluating the model's fault mode coverage capability, the model's fault mode coverage capability can be verified and evaluated based on these evaluation indicators and the model's processing results of the extracted samples. This approach helps improve the accuracy of the verification and evaluation results, thereby improving the actual performance of the model. Attached Figure Description

[0018] Figure 1 This is a flowchart of an adversarial verification and evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model, provided in one embodiment of this specification. Figure 2 This is a flowchart illustrating the processing procedure of an adversarial verification and evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model, provided in one embodiment of this specification. Figure 3 This is a schematic diagram of the structure of an adversarial verification and evaluation device for the boundary of fault mode coverage capability of a fault detection and diagnosis model, provided in one embodiment of this specification. Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0019] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0020] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0022] This specification provides an adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model. This specification also relates to an adversarial verification evaluation device for the fault mode coverage capability boundary of a fault detection and diagnosis model, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0023] Figure 1 The flowchart illustrates an adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model according to an embodiment of this specification, which specifically includes the following steps.

[0024] Step 102: Divide the obtained fault mode search domain into at least two subdomains, wherein the fault mode search domain is generated by a combination of parameters used to describe the fault modes of the target device.

[0025] Specifically, the fault mode search domain is formed by a combination of parameters that need to be examined to describe the fault modes.

[0026] The embodiments in this specification divide the current fault mode search domain evenly into A group, or K subfields, is denoted as (in, For the given hyperparameters, (where i represents the current round).

[0027] Step 104: Determine the number of samples to be extracted for each subdomain in the current round, and extract the corresponding target verification samples from the initial sample set according to the number of samples to be extracted.

[0028] Specifically, after determining the number of samples to be extracted for each target subdomain in the current i-th round, the corresponding target verification samples can be extracted from the initial sample set according to the number of samples extracted.

[0029] The embodiments in this specification can select corresponding verification samples from the fault diagnosis verification sample set collected in actual operation or the fault diagnosis verification sample set generated by simulation model, according to the determined fault mode search domain grouping and the sampling number of each group.

[0030] The initial sample set can be the validation set used during the training phase of the fault detection model under test, or other labeled datasets that meet the input requirements of the fault detection model under test and were not used during the training phase. More preferably, a dataset collected in the field after the model is deployed and run can be selected.

[0031] To further reduce computational load, the simulation model can be used to generate the corresponding number of target verification samples after the verification sample requirements are determined.

[0032] In one optional implementation, determining the number of samples to be extracted for each subdomain in the current round includes: Based on the accuracy of each verification sample belonging to the target subdomain in the (i-1)th round, determine the number of samples to be extracted for the target subdomain in the current i-th round, where i is a positive integer greater than or equal to 2, and the target subdomain is each of the at least two subdomains.

[0033] Furthermore, determining the number of samples to be extracted from the target subdomain in the current i-th round based on the accuracy of each verification sample belonging to the target subdomain in the (i-1)-th round includes: Based on the accuracy of each validation sample belonging to the target subdomain in the (i-1)th round, determine the first average accuracy corresponding to the target subdomain in the (i-1)th round; Determine the second average accuracy corresponding to the initial sample set; Based on the first average accuracy, the second average accuracy, and the preset basic sampling quantity, the number of samples to be extracted for the target subdomain in the current i-th round is determined.

[0034] Specifically, the number of samples drawn from each subdomain The accuracy is determined by the accuracy of each validation sample belonging to this subdomain in the previous round, i.e., the (i-1)th round. The specific calculation formula is as follows: in, Given the base sample size, These are proportional parameters, all of which are given parameters. This represents the average accuracy of all validation samples belonging to this subdomain in the previous round. Specifically, this applies when there are no validation samples belonging to this subdomain in the previous round. Take 0. This represents the average accuracy corresponding to the initial sample set.

[0035] Using the dynamic sample size adjustment method based on the logistic regression function, the more the validation samples in the target subdomain decrease the output accuracy of the tested fault detection model, the larger the sample size for that subdomain. Conversely, the more the validation samples in the target subdomain increase the output accuracy of the tested fault detection model, the smaller the sample size for that subdomain. This dynamic adjustment method allows for further reduction of the overall number of validation samples while ensuring a sufficient search range.

[0036] Specifically, during the first round of verification, the fault mode search domain is either the fault mode search domain formed by the initial sample set or a larger search domain that includes the original fault mode search domain mentioned above, which can be specified according to the verification needs.

[0037] Step 106: Process the target verification sample using the fault detection model to generate the corresponding fault detection result.

[0038] Specifically, the selected target validation sample set Input the fault detection model under test for inference. The fault detection model under test outputs the fault detection results for each validation sample in the current round's validation sample set.

[0039] The fault detection result includes at least: the model discrimination result of each target verification sample, the predicted probability of the discrimination result, the error with the sample label, and the intermediate process information generated during model inference (e.g., when the fault detection model under test is a deep learning model, the gradient information generated by model inference).

[0040] In one optional implementation, after generating the corresponding fault detection result, the method further includes: Based on the fault detection results, determine the overall accuracy of each target verification sample in the current i-th round; Determine whether the overall accuracy is less than a preset accuracy threshold; If so, then the step of verifying and evaluating the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators is executed.

[0041] Alternatively, determine whether the value of i in the current round is greater than or equal to the threshold of the number of items in the first round; If so, then the step of verifying and evaluating the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators is executed.

[0042] In addition, if the execution result of determining whether the overall accuracy is less than the preset accuracy threshold is no, or the execution result of determining whether the value of the current round i is greater than or equal to the first round quantity threshold is no, then the subdomain extraction ratio corresponding to the fault mode search domain is determined. According to the subdomain extraction ratio, subdomains are filtered in the at least two subdomains, and the target fault mode search domain is determined based on the subdomains obtained from the filtering. Increment i by 1, take the target fault mode search domain as the fault mode search domain, and return to execute the step of dividing the acquired fault mode search domain into at least two subdomains.

[0043] Furthermore, determining the subdomain extraction ratio corresponding to the fault mode search domain includes: If i is less than the second round quantity threshold, the initial extraction ratio is determined as the subdomain extraction ratio corresponding to the fault mode search domain; If i is greater than or equal to the second round quantity threshold, the subdomain extraction ratio corresponding to the fault mode search domain is determined based on the overall accuracy of each target verification sample in the current i-th round and the overall accuracy of each verification sample in the (i-1)-th round.

[0044] In one optional implementation, the fault detection results include continuous detection results; Accordingly, determining the overall accuracy of each target verification sample in the current i-th round based on the fault detection results includes: Determine the sample label corresponding to the first target verification sample in the current i-th round, wherein the first target verification sample is each of the target verification samples; Calculate the difference between the fault detection result corresponding to the first target verification sample and the sample label, and determine the first accuracy corresponding to the first target verification sample based on the difference; The overall accuracy of each target validation sample in the current i-th round is determined based on the first accuracy.

[0045] Alternatively, the fault detection result may include discrete detection results, which may include the probability of each fault category corresponding to the second target verification sample. Accordingly, determining the overall accuracy of each target verification sample in the current i-th round based on the fault detection results includes: The second accuracy corresponding to the second target verification sample is determined by the cross-entropy loss function and in combination with the probability and the number of fault categories, wherein the second target verification sample is each of the target verification samples; The overall accuracy corresponding to each target validation sample in the current i-th round is determined based on the second accuracy.

[0046] Specifically, after obtaining the fault detection results output by the fault detection model, the overall accuracy of each target validation sample in the current i-th round can be determined based on these results. .

[0047] Among them, determining the overall accuracy of each target verification sample in the current i-th round based on the fault detection results can first be calculated. Prediction accuracy of each target validation sample , Furthermore, considering the possibility that the fault detection model's output may include both discrete and continuous types, the current output type is selected. In the round Individual accuracy of a sample ( The methods include, but are not limited to, the following: For discrete outputs, the cross-entropy loss function can be used as a metric: in, For the number of categories, , For the sample Category The predicted probability.

[0048] For continuous outputs, a prediction error metric can be used: in, For predicted values, This is the tag value.

[0049] Based on the calculation of the prediction accuracy of each target validation sample, the average accuracy of each subdomain in the current i-th round can be calculated using the norm. and Overall accuracy : in, The norm of a vector of any type can be represented, typically either the 1-norm or the 2-norm. This is a vector composed of the prediction accuracy of the target validation samples belonging to this subdomain. , They are respectively and The number of elements contained in it.

[0050] Determine overall accuracy Then, the overall accuracy can be determined. Is it below the preset accuracy threshold? Alternatively, determine whether i has reached the maximum number of iterations T, i.e., determine whether i is greater than or equal to the threshold number of iterations in the first round T, where, And T are the specified parameters; if the overall accuracy Below the preset accuracy threshold If i is greater than or equal to the first round's quantity threshold T, then proceed to step 108; if the overall accuracy is... Greater than or equal to the preset accuracy threshold If i is less than the first round's quantity threshold T, then subsequent iteration rounds are required.

[0051] In practical applications, And T can be determined based on the accuracy requirements of the verification and evaluation results and the acceptable computational cost. The smaller the value of T, or the larger the value of T, the more accurate the verification and evaluation results will be, but the greater the computational cost will be.

[0052] In the embodiments described in this specification, the overall accuracy... Greater than or equal to the preset accuracy threshold If i is less than the first round's quantity threshold T, the subdomain extraction ratio corresponding to the aforementioned fault mode search domain can be determined. .

[0053] In practical applications, the proportion can be determined according to the following exponential decay function. .

[0054] in, This represents the initial sample proportion. To dynamically adjust the ratio, the first round begins. These are proportionality constants, all of which are given parameters.

[0055] The greater the overall output accuracy of the validation sample set in the previous round decreases, the fewer subdomains will be selected in subsequent rounds. This dynamic adjustment method allows for a further reduction in the total number of validation samples while ensuring a sufficient search range.

[0056] Determine the subdomain extraction ratio corresponding to the aforementioned fault mode search domain. Then, according to the extraction ratio of that subdomain, subdomain filtering is performed in at least two of the aforementioned subdomains to extract the validation sample set for the current i-th round. The subdomain to which each target validation sample belongs Each subdomain is considered, and their union is taken as the target fault mode search domain to be examined in the next round. Then, this target fault mode search domain can be used as the fault mode search domain, and the process returns to step 102 to evaluate and verify the fault detection model in the next round.

[0057] In practical applications, if it is necessary to obtain the lower bound verification evaluation results of the failure mode coverage capability, then the most difficult (i.e., the lowest average accuracy) failure mode in the current i-th round should be extracted. Subdomains; if an upper bound verification evaluation result of the fault mode coverage capability is required, then the subdomain with the best output performance (i.e., the highest average accuracy) in the current i-th round is extracted. Subdomains.

[0058] Step 108: Based on the fault detection results and preset evaluation indicators, verify and evaluate the fault mode coverage capability of the fault detection model, and generate corresponding verification and evaluation results.

[0059] Specifically, after obtaining the fault detection results output by the fault detection model, the fault mode coverage capability of the tested fault detection model can be analyzed and evaluated based on the fault detection results and the fault mode coverage capability evaluation index of the fault detection model.

[0060] Based on the fault detection results, evaluation metrics are designed to assess the fault mode coverage capability boundary of the fault detection model. Let the final round of validation sample set be denoted as . , , This is the union of the sample sets from all iterations except the initial sample set. The evaluation metric system for the embodiments in this specification is as follows: 1) Sample variability of boundary fault modes in, , yes The mean vector.

[0061] 2) Boundary Fault Mode Isolation Rate in, for The total number of samples in which faults were detected. for The number of samples that correctly identified all failure modes.

[0062] The above 1) and 2) evaluate the differences between samples in the final round of validation sample sets. When the goal of validation evaluation is to obtain a lower bound on the failure mode coverage capability, i.e., when selecting hard samples in the adversarial strategy, the evaluation metrics... The smaller the value of , the better the fault mode coverage capability of the tested fault detection model. Conversely, when the goal of validation evaluation is to obtain the upper bound of the fault mode coverage capability, that is, to select good samples in the adversarial strategy, the evaluation index is more important. The larger the value, the better the fault mode coverage capability of the fault detection model under test.

[0063] This specification's embodiments divide the fault mode search domain into several subdomains. Based on the average accuracy of the tested model for samples within a certain fault mode search domain, the subsequent sampling number for that group is determined. If the samples under a fault mode group cause a greater decrease in the accuracy of the tested model's output, then the sampling number for that group is larger. Based on the model output results and evaluation requirements, a certain proportion of groups in the fault mode search domain are selected to constitute the fault mode search domain for the next round. If a lower bound evaluation result for fault mode coverage is required, the most difficult group is selected. If an upper bound evaluation result for fault mode coverage is required, the group with the best output performance is selected. After the iterative verification process is completed, based on the specific requirements for evaluating the fault mode robustness of the fault detection and diagnosis model, a quantitative evaluation index system is established to comprehensively assess the fault mode robustness by comprehensively utilizing all historical test information generated during the iteration process. This specification's embodiments address the issue that fault detection and diagnosis models may encounter different fault mode conditions in actual operating environments, and propose a method for evaluating the fault mode coverage capability of fault detection and diagnosis models. This method has a wide range of applications and can be used to evaluate various fault detection and diagnosis models, including traditional machine learning models and deep learning models. Based on an adversarial iterative process, it dynamically adjusts the parameters for constructing validation samples according to the model's actual output, specifically reducing the search space, the number of queries, and the computational load. This method fully utilizes the process information of adversarial iteration to propose a dedicated evaluation index for the fault mode coverage capability of fault detection and diagnosis models. Compared with traditional general robustness evaluation indices, it is more suitable for evaluating the fault mode coverage capability of fault detection and diagnosis models, and the evaluation effect is more accurate.

[0064] This embodiment of the specification divides the acquired fault mode search domain into at least two subdomains, wherein the fault mode search domain is generated by a combination of parameters used to describe the fault modes of the target device; determines the number of samples to be extracted for each subdomain in the current round, and extracts corresponding target verification samples from the initial sample set according to the number of samples extracted; processes the target verification samples through a fault detection model to generate corresponding fault detection results; and verifies and evaluates the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators, generating corresponding verification and evaluation results. By dividing the fault mode search domain into subdomains, determining the number of samples to be extracted for each subdomain, and then extracting samples based on the number of samples extracted, and by designing evaluation indicators for evaluating the model's fault mode coverage capability, the model's fault mode coverage capability can be verified and evaluated based on these evaluation indicators and the model's processing results of the extracted samples. This approach helps improve the accuracy of the verification and evaluation results, thereby improving the actual performance of the model.

[0065] The following is in conjunction with the appendix Figure 2 The adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model is further explained. Figure 2 The present specification provides a flowchart of an adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model, which includes the following steps.

[0066] Step 202: Divide the obtained fault mode search domain into K groups.

[0067] Step 204: Based on the accuracy of each verification sample belonging to the target group in the (i-1)th round, determine the number of samples to be drawn for the target group in the current i-th round.

[0068] Step 206: Draw the corresponding target verification samples from the initial sample set according to the sample extraction quantity.

[0069] Specifically, the initial sample set can be the validation set used during the training phase of the fault detection model under test, or other labeled datasets that meet the input requirements of the fault detection model under test and were not used during the training phase. More preferably, a dataset collected in the field after the model is deployed and run can be selected.

[0070] To further reduce computational load, the corresponding target verification samples can be generated using a simulation model after the verification sample requirements are determined.

[0071] Step 208: Input the target verification sample into the fault detection model for processing, and obtain the fault detection result output by the fault detection model.

[0072] Step 210: Determine whether the overall accuracy corresponding to the target verification sample is less than the preset accuracy threshold, or determine whether i is greater than or equal to the first round quantity threshold.

[0073] If yes, proceed to step 218; otherwise, proceed to step 212.

[0074] Step 212: Determine the grouping extraction ratio corresponding to the fault mode search domain.

[0075] Step 214: Filter the K groups according to the grouping ratio, and determine the target fault mode search domain based on the filtered groups.

[0076] Step 216: Increment i by 1, and use the target fault mode search domain as the fault mode search domain.

[0077] Return to step 202.

[0078] Step 218: Design evaluation metrics for verifying and evaluating failure mode coverage capabilities.

[0079] Step 220: Based on the fault detection results and the evaluation indicators, verify and evaluate the fault mode coverage capability of the fault detection model, and generate the corresponding verification and evaluation results.

[0080] Corresponding to the above method embodiments, this specification also provides an embodiment of an adversarial verification and evaluation device for the fault mode coverage capability boundary of a fault detection and diagnosis model. Figure 3 This diagram illustrates a structural schematic of an adversarial verification and evaluation device for the fault mode coverage capability boundary of a fault detection and diagnosis model, provided in one embodiment of this specification. Figure 3 As shown, the device includes: The partitioning module 302 is configured to partition the acquired fault mode search domain into at least two subdomains, wherein the fault mode search domain is generated by a combination of parameters used to describe the fault modes of the target device. The extraction module 304 is configured to determine the number of samples to be extracted for each subdomain in the current round, and to extract the corresponding target verification samples from the initial sample set according to the number of samples to be extracted. The first generation module 306 is configured to process the target verification sample through a fault detection model to generate a corresponding fault detection result; The second generation module 308 is configured to verify and evaluate the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators, and generate corresponding verification and evaluation results.

[0081] Optionally, the extraction module 304 is further configured to: Based on the accuracy of each verification sample belonging to the target subdomain in the (i-1)th round, determine the number of samples to be extracted for the target subdomain in the current i-th round, where i is a positive integer greater than or equal to 2, and the target subdomain is each of the at least two subdomains.

[0082] Optionally, the extraction module 304 is further configured to: Based on the accuracy of each validation sample belonging to the target subdomain in the (i-1)th round, determine the first average accuracy corresponding to the target subdomain in the (i-1)th round; Determine the second average accuracy corresponding to the initial sample set; Based on the first average accuracy, the second average accuracy, and the preset basic sampling quantity, the number of samples to be extracted for the target subdomain in the current i-th round is determined.

[0083] Optionally, the adversarial verification and evaluation device for the fault mode coverage capability boundary of the fault detection and diagnosis model further includes a processing module configured as follows: Based on the fault detection results, determine the overall accuracy of each target verification sample in the current i-th round; Determine whether the overall accuracy is less than a preset accuracy threshold; If so, continue running the second generation module 308.

[0084] Optionally, the processing module is further configured to: Determine whether the value of the current round i is greater than or equal to the threshold number of the first round; If so, continue running the second generation module 308.

[0085] Optionally, the processing module is further configured to: If the result of determining whether the overall accuracy is less than the preset accuracy threshold is negative, then the subdomain extraction ratio corresponding to the fault mode search domain is determined. According to the subdomain extraction ratio, subdomains are filtered in the at least two subdomains, and the target fault mode search domain is determined based on the subdomains obtained from the filtering. i is incremented by 1, the target fault mode search domain is used as the fault mode search domain, and the partitioning module 302 continues to run.

[0086] Optionally, the processing module is further configured to: If the result of determining whether the value of the current round i is greater than or equal to the first round quantity threshold is negative, then the subdomain extraction ratio corresponding to the fault mode search domain is determined. According to the subdomain extraction ratio, subdomains are filtered in the at least two subdomains, and the target fault mode search domain is determined based on the subdomains obtained from the filtering. i is incremented by 1, the target fault mode search domain is used as the fault mode search domain, and the partitioning module 302 continues to run.

[0087] The above is a schematic scheme of an adversarial verification and evaluation device for the boundary of fault mode coverage capability of a fault detection and diagnosis model according to this embodiment. It should be noted that the technical solution of this adversarial verification and evaluation device for the boundary of fault mode coverage capability of a fault detection and diagnosis model belongs to the same concept as the technical solution of the adversarial verification and evaluation method for the boundary of fault mode coverage capability of a fault detection and diagnosis model described above. Details not described in detail in the technical solution of the adversarial verification and evaluation device for the boundary of fault mode coverage capability of a fault detection and diagnosis model can be found in the description of the technical solution of the adversarial verification and evaluation method for the boundary of fault mode coverage capability of a fault detection and diagnosis model described above.

[0088] Figure 4 A structural block diagram of a computing device 400 according to one embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0089] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0090] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0091] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 400 can also be a mobile or stationary server.

[0092] The processor 420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model.

[0093] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model described above.

[0094] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model described above.

[0095] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model described above.

[0096] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, the computer performs the steps of the above-described adversarial verification and evaluation method for the boundary of fault mode coverage capability of a fault detection and diagnosis model.

[0097] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solution of the adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model described above. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the adversarial verification evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model described above.

[0098] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0099] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0100] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0101] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0102] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An adversarial verification and evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model, comprising: The acquired fault mode search domain is divided into at least two subdomains, wherein the fault mode search domain is generated by a combination of parameters used to describe the fault modes of the target device. Determine the number of samples to be extracted for each subdomain in the current round, and extract the corresponding target verification samples from the initial sample set according to the number of samples to be extracted; The target verification sample is processed by a fault detection model to generate corresponding fault detection results; Based on the fault detection results and preset evaluation indicators, the fault mode coverage capability of the fault detection model is verified and evaluated, and corresponding verification and evaluation results are generated.

2. The adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model according to claim 1, wherein determining the number of samples extracted for each subdomain in the current round includes: Based on the accuracy of each verification sample belonging to the target subdomain in the (i-1)th round, determine the number of samples to be extracted for the target subdomain in the current i-th round, where i is a positive integer greater than or equal to 2, and the target subdomain is each of the at least two subdomains.

3. The adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model according to claim 2, wherein determining the number of samples to be extracted from the target subdomain in the current i-th round based on the accuracy of each verification sample belonging to the target subdomain in the (i-1)-th round includes: Based on the accuracy of each validation sample belonging to the target subdomain in the (i-1)th round, determine the first average accuracy corresponding to the target subdomain in the (i-1)th round; Determine the second average accuracy corresponding to the initial sample set; Based on the first average accuracy, the second average accuracy, and the preset basic sampling quantity, the number of samples to be extracted for the target subdomain in the current i-th round is determined.

4. The adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model according to claim 1, further comprising, after generating the corresponding fault detection result: Based on the fault detection results, determine the overall accuracy of each target verification sample in the current i-th round; Determine whether the overall accuracy is less than a preset accuracy threshold; If so, then the step of verifying and evaluating the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators is executed.

5. The adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model according to claim 1, further comprising, after generating the corresponding fault detection result: Determine whether the value of the current round i is greater than or equal to the threshold number of the first round; If so, then the step of verifying and evaluating the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators is executed.

6. The adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model according to claim 4 further includes: If the result of determining whether the overall accuracy is less than the preset accuracy threshold is negative, then the subdomain extraction ratio corresponding to the fault mode search domain is determined. According to the subdomain extraction ratio, subdomains are filtered in the at least two subdomains, and the target fault mode search domain is determined based on the subdomains obtained from the filtering. Increment i by 1, take the target fault mode search domain as the fault mode search domain, and return to execute the step of dividing the acquired fault mode search domain into at least two subdomains.

7. The adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model according to claim 5 further includes: If the result of determining whether the value of the current round i is greater than or equal to the first round quantity threshold is negative, then the subdomain extraction ratio corresponding to the fault mode search domain is determined. According to the subdomain extraction ratio, subdomains are filtered in the at least two subdomains, and the target fault mode search domain is determined based on the subdomains obtained from the filtering. Increment i by 1, take the target fault mode search domain as the fault mode search domain, and return to execute the step of dividing the acquired fault mode search domain into at least two subdomains.

8. An adversarial verification and evaluation device for the fault mode coverage capability boundary of a fault detection and diagnosis model, comprising: The partitioning module is configured to divide the acquired fault mode search domain into at least two subdomains, wherein the fault mode search domain is generated by a combination of parameters used to describe the fault modes of the target device. The extraction module is configured to determine the number of samples to be extracted for each subdomain in the current round, and to extract the corresponding target verification samples from the initial sample set according to the number of samples to be extracted. The first generation module is configured to process the target verification sample through a fault detection model to generate a corresponding fault detection result; The second generation module is configured to verify and evaluate the fault mode coverage capability of the fault detection model based on the fault detection results and preset evaluation indicators, and generate corresponding verification and evaluation results.

9. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the adversarial verification and evaluation method for the fault mode coverage capability boundary of the fault detection and diagnosis model as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the adversarial verification evaluation method for the fault mode coverage capability boundary of a fault detection and diagnosis model as described in any one of claims 1 to 7.

Images