A Failure Mechanism Analysis Method for Intelligent Recognition Systems Based on Do Operator Intervention

Abstract

This invention addresses intelligent recognition systems and proposes a failure mechanism analysis method based on Do operator intervention. It comprises four main steps: (1) constructing a task scenario for the intelligent recognition system and defining the system's recognition error rate; (2) analyzing key influencing factors in the task scenario, determining the "influencing factors-evaluation parameters-feature parameters" data structure, and constructing a factor-based influence model for the system's recognition error rate; (3) creating a test dataset containing different defects and obtaining test data through simulation experiments; (4) processing the test data and normalizing the evaluation parameter data of each influencing factor; and (5) building a causal inference model based on Do operator intervention, estimating the average causal effect, and conducting failure mechanism analysis.

Description

Technical Field

[0001] This invention addresses intelligent recognition systems and proposes a failure mechanism analysis method based on the Do operator intervention. This method analyzes key influencing factors that may lead to system recognition failure throughout the entire task process, constructs a factor influence model of the system's recognition error rate, generates a defective test dataset to obtain experimental data, constructs a causal graph, builds a causal inference model based on the Do operator intervention, and uses the Do operator inference rule to estimate the average causal effect of influencing factors on system recognition failure. This allows for a better identification of the relative strength of different influencing factors on the system's recognition results, providing a reference for failure mechanism analysis of intelligent recognition systems. This invention belongs to the field of artificial intelligence. Background Technology

[0002] As the accuracy and intelligence levels of biometric technology continue to improve, intelligent identification technology is gradually occupying an important position in the field of biometric recognition, with wide applications in finance, criminal investigation, and the judiciary. At the same time, the security and reliability of intelligent identification systems during missions are receiving increasing attention, necessitating a comprehensive and robust method for analyzing system failure mechanisms to assess the impact of investigated factors on system failure and provide a basis for the design and optimization of intelligent identification systems.

[0003] Intelligent recognition systems are often affected by various factors during the execution of recognition tasks. During the intelligent model training phase, the size of the training dataset, the language used, the general model architecture employed, and different training strategies all influence the system's recognition performance. During the system deployment phase, the length of registered and test samples, noise levels, and acquisition methods also affect the recognition results. Currently, research on failure mechanism analysis for intelligent recognition systems is not comprehensive enough, lacking a failure mechanism analysis method that covers all task stages. Therefore, this invention proposes a failure mechanism analysis method for intelligent recognition systems based on Do operator intervention. By analyzing the task profile in the intelligent recognition system's recognition task, this method comprehensively analyzes the key influencing factors at each task stage, selects appropriate evaluation parameters for each influencing factor, creates a test dataset containing different defects, conducts simulation tests to obtain experimental data, uses a causal graph model to represent the causal relationships between variables, builds a causal inference model based on Do operator intervention, and uses Do operator intervention and Do operator inference rules to estimate the average causal effect of influencing factors on system recognition failure, thereby identifying the strength of the influence of different influencing factors on the system's recognition results. This provides a reference for the failure mechanism analysis and design optimization of intelligent voiceprint recognition systems. Summary of the Invention

[0004] This invention provides a failure mechanism analysis method for an intelligent identification system based on Do operator intervention, see [link to relevant documentation]. Figure 1The main steps of this invention are as follows:

[0005] Step 1: Construct the task scenarios for the intelligent recognition system and define the system's recognition error rates;

[0006] The task process of the intelligent recognition system is divided into three stages: model training, feature registration, and identity recognition. The false positive rate is defined as the probability that a false positive sample is accepted when the system performs the recognition task, and its calculation formula is:

[0007]

[0008] Where FP is the number of false positive samples, TN is the number of true negative samples, and FAR is the false alarm rate.

[0009] The false negative rate is defined as the probability that a true negative sample is rejected, and its calculation formula is as follows:

[0010]

[0011] Where FN is the number of false negatives, TP is the number of true positives, and FRR is the false negative rate.

[0012] The error rate at which the false positive rate equals the false negative rate is defined as the system identification error rate, and its calculation formula is as follows:

[0013] FAR = FRR = EER

[0014] Here, EER represents the equal error rate of the system's task identification.

[0015] Step 2: Analyze the key influencing factors in the task scenario, determine the data structure of "influencing factors-evaluation parameters-feature parameters", and construct a factor influence model for the error rate of system recognition, etc.

[0016] Step 1: Analyze the key influencing factors in each task scenario;

[0017] Analyze the key influencing factors in the three task stages of intelligent recognition system: model training, feature registration, and identity recognition, and obtain the set of influencing factors, denoted as . Where i = {1, 2, 3}, m ij Let n represent the j-th influencing factor in the i-th stage. i This represents the number of influencing factors in stage i.

[0018] Step 2: Construct a data structure of "influencing factors - evaluation parameters - characteristic parameters";

[0019] Analyze the various influencing factors obtained in Step 1 of Step 2, select a suitable quantitative evaluation parameter that can characterize the influencing factor and its changing trend, and obtain the set of evaluation parameters for the influencing factor, denoted as . Where i = {1, 2, 3}, p ij Indicates influencing factor m ij Evaluation parameters.

[0020] Based on the characteristics of parameter values ​​and whether the values ​​are ordered, evaluation parameters are classified into numerical parameters, categorical parameters, and degree parameters. The classification rules for these three types of parameters are as follows:

[0021] (1) Classification rules for numerical parameters: values ​​are continuous real numbers;

[0022] (2) Classification rules for categorical parameters: can be quantified into discrete real numbers, and the values ​​are unordered;

[0023] (3) Classification rules for degree-type parameters: can be quantified into discrete real numbers, and the values ​​are ordered.

[0024] Analyze each evaluation parameter and select one or more characteristic parameters according to the different parameter categories:

[0025] (1) For numerical parameters, their values ​​are defined as characteristic parameters, and the number of characteristic parameters is 1;

[0026] (2) For categorical parameters p, the number of feature parameters is determined using a binary encoding method. First, the category of parameter p is encoded, denoted as pi. Type ={1,2,...,n p}, where p Type n represents the set of category codes for parameter p. p This represents the total number of categories for the parameter; then, the category encoding of the parameter is converted into a binary representation, with each binary bit representing a feature parameter. The rule for determining the number of binary bits based on the total number of parameter categories is as follows:

[0027] ①n p ≤2: The number of binary bits is 1, and the number of feature parameters is 1;

[0028] ②2<n p ≤4: The number of binary bits is 2, and the number of feature parameters is 2;

[0029] ③4<n p ≤8: The number of binary bits is 3, and the number of feature parameters is 3.

[0030] (3) For degree-type parameters, numerical values ​​are generated using an interval mapping method. The degree-type parameters are numerically generated according to the parameter value order and relative size, and mapped to the interval (0,1).

[0031] Based on the above analysis, a set of feature parameters is obtained, denoted as C = {c p |p∈P i ;i=1,2,3}, where c pThe characteristic parameter represents the evaluation parameter p.

[0032] Complete the construction of the "Influencing Factors-Evaluation Parameters-Feature Parameters" data structure for the intelligent recognition system, namely the "MPC" data structure, where M = {M i}, i={1,2,3},P={P i}, i = {1, 2, 3}.

[0033] Step 3: Construct a model of factors influencing the error rate of system identification, etc.;

[0034] A factor-driven model is constructed to influence the error rate of the system recognition, simulating the impact of key influencing factors on the recognition task results of the intelligent recognition system. The model is as follows:

[0035] EER=f(C)+g(U EER )

[0036] Where f(C) represents the influence of feature parameter C on the system's recognition error rate, g(U EER ) represents random influencing factor U EER Impact on error rates in system recognition, etc.

[0037] Step 3: Create a test dataset containing different defects and obtain test data through simulation experiments;

[0038] Step 1: Adjust the values ​​of the characteristic parameters of each influencing factor, conduct preliminary experiments, and obtain the range of zero-defect test data.

[0039] For influencing factor m 11 Its evaluation parameter is p 11 The characteristic parameters are Keeping the values ​​of the characteristic parameters of other influencing factors constant, adjust The value of m is recorded to determine the error rate of the system's recognition task and obtain the influencing factor m. 11 Feature parameters The range of values ​​that have no impact on the error rate of system recognition, etc., is denoted as . in Representing characteristic parameters The minimum value that has no impact on the error rate of system recognition, etc. Indicates the characteristic parameter c p11 The maximum value that has no impact on the error rate of system recognition, etc.

[0040] Repeat the above process to obtain the set of value ranges of all influencing factors' feature parameters that have no impact on the system's recognition error rate, denoted as . Based on this, the range of zero-defect test data values ​​is defined as follows:

[0041] Step 2: Create a test dataset;

[0042] The test dataset consists of three subsets: the intelligent model set, the registration set, and the identification set. The registration set and the identification set each include samples with defects and samples without defects, respectively.

[0043] (1) Creating an intelligent model set: By modifying the feature parameter values ​​of influencing factors during the model training phase, different intelligent recognition models are obtained through training. An intelligent model set is created, including N1 intelligent models.

[0044] (2) Create a defective sample set: Modify the values ​​of relevant feature parameters for each sample data. When the values ​​of q feature parameters are not within the range R0 of the zero-defect test data, it is considered that q defects have been injected into the test data. Create a defective sample set, including: N2 registered samples with defects and N3 identified samples with defects.

[0045] (3) Create a defect-free sample set: Modify the values ​​of the relevant feature parameters for each sample data to ensure that all feature parameter values ​​are within the zero-defect test data range R0, and are considered defect-free test data. Create a defect-free sample set, including: N'2 defect-free registration set samples and N3' defect-free identification set samples.

[0046] (4) Create a test dataset: merge the intelligent model set, the defective sample set and the non-defective sample set to obtain a test dataset, which includes three subsets: intelligent model set (including N1 intelligent models), registration set (including N2 defective registration set samples and N'2 non-defective registration set samples) and identification set (including N3 defective identification set samples and N3' non-defective identification set samples).

[0047] Step 3: Obtain test data through simulation experiments;

[0048] The simulation experiment was conducted using the test dataset obtained in step 2 of step 3. The simulation test steps are as follows:

[0049] (1) Select an intelligent recognition model from the set of intelligent models;

[0050] (2) Select M registration samples with the same feature parameter values ​​from the registration set and register them in the selected intelligent recognition model, where M≤N2+N'2, and build an intelligent recognition system.

[0051] (3) Randomly select O identification samples with the same feature parameter values ​​from the identification set for identification testing, where O≤N3+N3', and record the system identification error rate. Obtain one test result data, which includes two parts: the feature parameter values ​​of all influencing factors and the system identification error rate.

[0052] (4) Repeat steps (1) to (3) Z times, and finally obtain a total of Z test result data.

[0053] Step 4: Process the test data and normalize the evaluation parameter data of each influencing factor;

[0054] Categorical parameters, after binary encoding, already have values ​​within the range (0,1), requiring no further normalization. Degree parameters are numerically represented and mapped to the range (0,1) using an interval mapping method, also requiring no further normalization. The normalization formula for the feature parameters of numerical parameters is as follows:

[0055]

[0056] Where c' is the normalized data of data c, c max c min These are the maximum and minimum values ​​of data c in the test results data, respectively.

[0057] The system recognition error rate (EER) ranges from [0, 0.5]. After normalization, its calculation formula is as follows:

[0058] EER'=2×EER

[0059] Obtain the test data matrix Where D i This represents the i-th test data. This indicates that each test data point includes normalized data of characteristic parameters of influencing factors in all stages of the intelligent recognition system and the system's recognition error rate.

[0060] Step 5: Construct a causal inference model based on Do operator intervention, estimate the average causal effect, and conduct failure mechanism analysis;

[0061] Step 1: Construct a causal graph relating influencing factors to the error rate of system identification, etc.

[0062] One feature parameter corresponds to one node in the causal graph, and the data EER' corresponds to the random influencing factor U. EER Each node in the causal graph corresponds to a node in the causal graph. The set of nodes in the causal graph, V, is defined as denoted as . Where K represents the sum of characteristic parameters of the influencing factors under investigation. Based on the factor influence model of system identification error rate, the causal relationships between nodes are analyzed to obtain the causal relationship set E = {(V i V j )}, where (V i V j ) represents V i For V j The reason, Vj For V i The result is that a causal graph G(V,E) is constructed to connect influencing factors with the error rate of system identification, etc.

[0063] Step 2: Define the average causal effect;

[0064] The average causal effect is used to quantify the impact of changes in the values ​​of target influencing factors on the error rate of system identification, etc. The calculation formula is as follows:

[0065] ACE = E(V) EER' |Do(V=v))-E(V EER' |Do(V=v'))

[0066] Where ACE is the average causal effect, E(V) EER' |Do(V=v)) represents intervening in node V such that when the value of V is v, node V... EER' The mean of the values, U EER It is a random factor, and its node mean is 0.

[0067] According to the Do operator inference rule, the above formula is simplified and transformed into a formula for calculating the causal effect between variables using observational data. The calculation formula is as follows:

[0068]

[0069] Among them, V a It is all relatively ordered pairs of nodes (V,V) EER' The set of variables that satisfy the backdoor criterion, i.e., V a Given values ​​for all variables within the set, node V and node V EER' They are independent of each other.

[0070] Step 3: Construct a causal inference model based on Do operator intervention;

[0071] A causal inference model based on Do operator intervention is constructed. The model's input consists of a causal graph between influencing factors and system identification error rates, along with simulation test data; the output is the average causal effect of the target influencing factor on system identification error rates. The model's operation is as follows: Do intervention is performed on the target influencing factor, altering the causal relationship between the target influencing factor and other factors. By updating the causal graph between the target influencing factor and system identification error rates, the average causal effect is estimated using the Do operator inference rule, thus obtaining the average causal effect of the target influencing factor on system identification error rates.

[0072] Step 4: Conduct a robustness test to decide whether to accept the results;

[0073] Robustness testing is performed using random common factor refutation. Independent random variables are added to the dataset and the causal effect value is estimated again. If the estimated result after refutation is not significantly different from the original result, the result is accepted and the process proceeds to step 5. Otherwise, the result is rejected and step 1 of step 5 is repeated.

[0074] Step 5: Analyze the failure mechanism of the intelligent recognition system;

[0075] Based on the results obtained in step 3, the influence of different influencing factors on the error rate of the system recognition is identified by comparing the average causal effect size, thereby determining the failure mechanism of the intelligent recognition system. Attached Figure Description

[0076] Figure 1 A method for analyzing the failure mechanism of an intelligent identification system based on Do operator intervention.

[0077] Figure 2 Causal inference model based on Do operator intervention

[0078] Figure 3 Cause-and-effect graph of influencing factors and system recognition error rate Specific implementation methods

[0079] Step 1: Construct the task scenarios for the intelligent recognition system and define the system's recognition error rates;

[0080] The task process of the intelligent recognition system is divided into three stages: model training, feature registration, and identity recognition. The false positive rate is defined as the probability that a false positive sample is accepted when the system performs the recognition task, and its calculation formula is:

[0081]

[0082] Where FP is the number of false positive samples, TN is the number of true negative samples, and FAR is the false alarm rate.

[0083] The false negative rate is defined as the probability that a true negative sample is rejected, and its calculation formula is as follows:

[0084]

[0085] Where FN is the number of false negatives, TP is the number of true positives, and FRR is the false negative rate.

[0086] The error rate at which the false positive rate equals the false negative rate is defined as the system identification error rate, and its calculation formula is as follows:

[0087] FAR = FRR = EER

[0088] Here, EER represents the equal error rate of the system's task identification.

[0089] [Example] The task process of an intelligent voiceprint recognition system is divided into three stages: model training, feature registration, and identity recognition. During the identity recognition task, as the set similarity threshold increases, the false alarm rate (FAR) decreases while the false positive rate (FRR) increases. Assuming the similarity threshold is set to 0.6, the false alarm rate and false positive rate are equal, both at 8%. Then, the system's overall recognition error rate is:

[0090] EER = FAR = FRR = 8%

[0091] Step 2: Analyze the key influencing factors in the task scenario, construct a data structure of "influencing factors-evaluation parameters-feature parameters", and build a factor influence model for the error rate of system recognition, etc.

[0092] Step 1: Analyze the key influencing factors in each task scenario;

[0093] Analyze the key influencing factors in the three task stages of intelligent recognition system: model training, feature registration, and identity recognition, and obtain the set of influencing factors, denoted as . Where i = {1, 2, 3}, m ij Let n represent the j-th influencing factor in the i-th stage. i This represents the number of influencing factors in stage i.

[0094] Step 2: Construct a data structure of "influencing factors - evaluation parameters - characteristic parameters";

[0095] Analyze the various influencing factors obtained in Step 1 of Step 2, select a suitable quantitative evaluation parameter that can characterize the influencing factor and its changing trend, and obtain the set of evaluation parameters for the influencing factor, denoted as . Where i = {1, 2, 3}, p ij Indicates influencing factor m ij Evaluation parameters.

[0096] Based on the characteristics of parameter values ​​and whether the values ​​are ordered, evaluation parameters are classified into numerical parameters, categorical parameters, and degree parameters. The classification rules for these three types of parameters are as follows:

[0097] (1) Classification rules for numerical parameters: values ​​are continuous real numbers;

[0098] (2) Classification rules for categorical parameters: can be quantified into discrete real numbers, and the values ​​are unordered;

[0099] (3) Classification rules for degree-type parameters: can be quantified into discrete real numbers, and the values ​​are ordered.

[0100] Analyze each evaluation parameter and select one or more characteristic parameters according to the different parameter categories:

[0101] (1) For numerical parameters, their values ​​are defined as characteristic parameters, and the number of characteristic parameters is 1;

[0102] (2) For categorical parameters p, the number of feature parameters is determined using a binary encoding method. First, the category of parameter p is encoded, denoted as pi. Type ={1,2,...,n p}, where p Type n represents the set of category codes for parameter p. p This represents the total number of categories for the parameter; then, the category encoding of the parameter is converted into a binary representation, with each binary bit representing a feature parameter. The rule for determining the number of binary bits based on the total number of parameter categories is as follows:

[0103] ①n p ≤2: The number of binary bits is 1, and the number of feature parameters is 1;

[0104] ②2<n p ≤4: The number of binary bits is 2, and the number of feature parameters is 2;

[0105] ③4<n p ≤8: The number of binary bits is 3, and the number of feature parameters is 3.

[0106] (3) For degree-type parameters, numerical values ​​are generated using an interval mapping method. The degree-type parameters are numerically generated according to the parameter value order and relative size, and mapped to the interval (0,1).

[0107] Based on the above analysis, a set of feature parameters is obtained, denoted as C = {c p |p∈P i ;i=1,2,3}, where c p The characteristic parameter represents the evaluation parameter p.

[0108] Complete the construction of the "Influencing Factors-Evaluation Parameters-Feature Parameters" data structure for the intelligent recognition system, namely the "MPC" data structure, where M = {M i}, i={1,2,3},P={P i}, i = {1, 2, 3}.

[0109] Step 3: Construct a model of factors influencing the error rate of system identification, etc.;

[0110] A factor-driven model is constructed to influence the error rate of the system recognition, simulating the impact of key influencing factors on the recognition task results of the intelligent recognition system. The model is as follows:

[0111] EER=f(C)+g(U EER )

[0112] Where f(C) represents the influence of feature parameter C on the system's recognition error rate, g(U EER ) represents random influencing factor U EER Impact on error rates in system recognition, etc.

[0113] [Example] Analyze the key influencing factors in the three task stages of an intelligent voiceprint recognition system: model training, feature registration, and identity recognition, and obtain a set of influencing factors:

[0114] M1={m 11 =Size of training dataset, m 12 =Intelligent Model Architecture}

[0115] M2={m 21 =Registered voice recording duration}

[0116] M3={m 31 =Speech recognition duration, m 32 = Identify background noise in speech, m 33 =Identify voice channel}

[0117] The selected training dataset is CN-Celeb, with a total duration of T; the selected intelligent model architectures include the CAM++ model architecture and the ECAPA-TDNN model architecture.

[0118] The various influencing factors obtained from the analysis are analyzed, and appropriate quantitative evaluation parameters that can characterize the influencing factor and its changing trend are selected to obtain a set of evaluation parameters for the influencing factors:

[0119] P1={p 11 = Relative duration of training dataset, p 12 =Intelligent Model Architecture Type}

[0120] P2={p 21 =Registration Sample Duration}

[0121] P3={p 31 = Sample recognition time, p 32 =Signal-to-noise ratio of the identified samples, p 33 =Whether the identification sample and the registered sample channel are the same}

[0122] The evaluation parameters were analyzed, and one or more feature parameters were selected according to the parameter category and specific characteristics. The feature parameter selection results of the intelligent voiceprint recognition system are shown in Table 1.

[0123] Table 1. Results of Feature Parameter Selection for Intelligent Voiceprint Recognition System

[0124]

[0125]

[0126] Obtain the feature parameter set C = {c p |p∈P i ;i=1,2,3}, complete the construction of the "influencing factors-evaluation parameters-feature parameters" data structure of the intelligent recognition system, namely the "MPC" data structure.

[0127] A factor-driven model is constructed to influence the error rate of the system recognition, simulating the impact of key influencing factors on the recognition task results of the intelligent recognition system. The model is as follows:

[0128] EER=f(C)+g(U EER )

[0129] Step 3: Create a test dataset containing different defects and obtain test data through simulation experiments;

[0130] Step 1: Adjust the values ​​of the characteristic parameters of each influencing factor, conduct preliminary experiments, and obtain the range of zero-defect test data.

[0131] For influencing factor m 11 Its evaluation parameter is p 11 The characteristic parameters are Keeping the values ​​of the characteristic parameters of other influencing factors constant, adjust The value of m is recorded to determine the error rate of the system's recognition task and obtain the influencing factor m. 11 Feature parameters The range of values ​​that have no impact on the error rate of system recognition, etc., is denoted as . in Representing characteristic parameters The minimum value that has no impact on the error rate of system recognition, etc. Representing characteristic parameters The maximum value that has no impact on the error rate of system recognition, etc.

[0132] Repeat the above process to obtain the set of value ranges of all influencing factors' feature parameters that have no impact on the system's recognition error rate, denoted as . Based on this, the range of zero-defect test data values ​​is defined as follows:

[0133] Step 2: Create a test dataset;

[0134] The test dataset consists of three subsets: the intelligent model set, the registration set, and the identification set. The registration set and the identification set each include samples with defects and samples without defects, respectively.

[0135] (1) Creating an intelligent model set: By modifying the feature parameter values ​​of influencing factors during the model training phase, different intelligent recognition models are obtained through training. An intelligent model set is created, including N1 intelligent models.

[0136] (2) Create a defective sample set: Modify the values ​​of relevant feature parameters for each sample data. When the values ​​of q feature parameters are not within the range R0 of the zero-defect test data, it is considered that q defects have been injected into the test data. Create a defective sample set, including: N2 registered samples with defects and N3 identified samples with defects.

[0137] (3) Create a defect-free sample set: Modify the values ​​of the relevant feature parameters for each sample data to ensure that all feature parameter values ​​are within the zero-defect test data range R0, and are considered defect-free test data. Create a defect-free sample set, including: N'2 defect-free registration set samples and N3' defect-free identification set samples.

[0138] (4) Create a test dataset: merge the intelligent model set, the defective sample set and the non-defective sample set to obtain a test dataset, which includes three subsets: intelligent model set (including N1 intelligent models), registration set (including N2 defective registration set samples and N'2 non-defective registration set samples) and identification set (including N3 defective identification set samples and N3' non-defective identification set samples).

[0139] Step 3: Obtain test data through simulation experiments;

[0140] The simulation experiment was conducted using the test dataset obtained in step 2 of step 3. The simulation test steps are as follows:

[0141] (1) Select an intelligent recognition model from the set of intelligent models;

[0142] (2) Select M registration samples with the same feature parameter values ​​from the registration set and register them in the selected intelligent recognition model, where M≤N2+N'2, and build an intelligent recognition system.

[0143] (3) Randomly select O identification samples with the same feature parameter values ​​from the identification set for identification testing, where O≤N3+N3', and record the system identification error rate. Obtain one test result data, which includes two parts: the feature parameter values ​​of all influencing factors and the system identification error rate.

[0144] (4) Repeat steps (1) to (3) Z times, and finally obtain a total of Z test result data.

[0145] [Example] Adjust the values ​​of the characteristic parameters of each influencing factor, conduct preliminary experiments, and obtain the range of zero-defect test data values:

[0146]

[0147] The test dataset consists of three subsets: the intelligent model set (containing 4 intelligent models), the registration set (containing 40 defective registration set samples and 320 defect-free registration set samples), and the identification set (containing 12,000 defective identification set samples and 2,400 defect-free identification set samples).

[0148] The obtained test dataset was used to conduct simulation experiments. The simulation test steps are as follows:

[0149] (1) Select an intelligent recognition model from the set of intelligent models;

[0150] (2) Select 10 registered samples with the same feature parameter values ​​from the registration set and register them in the selected intelligent recognition model to build an intelligent recognition system;

[0151] (3) Select 400 identification samples with the same feature parameter values ​​from the identification set for identification testing, and record the system identification error rate. Obtain one test result data, which includes two parts: the feature parameter values ​​of all influencing factors and the system identification error rate.

[0152] (4) Repeat steps (1) to (3) 144 times, and finally obtain a total of 144 test result data.

[0153] The following table shows some of the test results:

[0154] Table 2 Test Results Data (Partial)

[0155]

[0156] Step 4: Process the test data and normalize the evaluation parameter data of each influencing factor;

[0157] Categorical parameters, after binary encoding, already have values ​​within the range (0,1), requiring no further normalization. Degree parameters are numerically represented and mapped to the range (0,1) using an interval mapping method, also requiring no further normalization. The normalization formula for the feature parameters of numerical parameters is as follows:

[0158]

[0159] Where c' is the normalized data of data c, c max c min These are the maximum and minimum values ​​of data c in the test results data, respectively.

[0160] The system recognition error rate (EER) ranges from [0, 0.5]. After normalization, its calculation formula is as follows:

[0161] EER'=2×EER

[0162] Obtain the test data matrix Where D i This represents the i-th test data. This indicates that each test data point includes normalized data of characteristic parameters of influencing factors in all stages of the intelligent recognition system and the system's recognition error rate.

[0163] [Example] Normalizing the characteristic parameters of numerical parameters:

[0164] (1) Feature parameters (Relative duration of training dataset) normalization:

[0165]

[0166] (2) Feature parameters (Registration sample duration) normalized:

[0167]

[0168] (3) Feature parameters (Sample identification time) normalization:

[0169]

[0170] (4) Feature parameters (Signal-to-noise ratio of identified samples) normalization:

[0171]

[0172] (5) System recognition error rate EER normalization:

[0173] EER'=2×EER

[0174] The partially normalized test results are shown in the table below:

[0175] Table 3. Normalized test results (partial)

[0176]

[0177]

[0178] Step 5: Build a causal inference model based on Do operator intervention, estimate the average causal effect, and conduct failure mechanism analysis;

[0179] Step 1: Construct a causal graph relating influencing factors to the error rate of system identification, etc.

[0180] One feature parameter corresponds to one node in the causal graph, and the data EER' corresponds to the random influencing factor U. EER Each node in the causal graph corresponds to a node in the causal graph. The set of nodes in the causal graph, V, is defined as denoted as . Where K represents the sum of characteristic parameters of the influencing factors under investigation. Based on the factor influence model of system identification error rate, the causal relationships between nodes are analyzed to obtain the causal relationship set E = {(V i V j )}, where (V i V j ) represents V i For V j The reason, V j For V i The result is that a causal graph G(V,E) is constructed to connect influencing factors with the error rate of system identification, etc.

[0181] Step 2: Define the average causal effect;

[0182] The average causal effect is used to quantify the impact of changes in the values ​​of target influencing factors on the error rate of system identification, etc. The calculation formula is as follows:

[0183] ACE = E(V) EER' |Do(V=v))-E(V EER' |Do(V=v'))

[0184] Where ACE is the average causal effect, E(V) EER' |Do(V=v)) represents intervening in node V such that when the value of V is v, node V... EER' The mean of the values, U EER It is a random factor, and its node mean is 0.

[0185] According to the Do operator inference rule, the above formula is simplified and transformed into a formula for calculating the causal effect between variables using observational data. The calculation formula is as follows:

[0186]

[0187] Among them, V a It is all relatively ordered pairs of nodes (V,V) EER' The set of variables that satisfy the backdoor criterion, i.e., V a Given values ​​for all variables within the set, node V and node V EER' They are independent of each other.

[0188] Step 3: Construct a causal inference model based on Do operator intervention;

[0189] Constructing a causal inference model based on Do operator intervention, such as Figure 2As shown, the model's input consists of a causal graph between influencing factors and the system's error rate (e.g., system identification error rate), and simulation test data; the output is the average causal effect of the target influencing factor on the system identification error rate. The model's operation is as follows: a Do intervention is performed on the target influencing factor, altering the causal relationship between the target influencing factor and other factors. The causal graph between the target influencing factor and the system identification error rate is updated, and the average causal effect is estimated using the Do operator inference rule, thus obtaining the average causal effect of the target influencing factor on the system identification error rate.

[0190] Step 4: Conduct a robustness test to decide whether to accept the results;

[0191] Robustness testing is performed using random common factor refutation. Independent random variables are added to the dataset and the causal effect value is estimated again. If the estimated result after refutation is not significantly different from the original result, the result is accepted and the process proceeds to step 5. Otherwise, the result is rejected and step 1 of step 5 is repeated.

[0192] Step 5: Analyze the failure mechanism of the intelligent recognition system;

[0193] Based on the results obtained in step 3, the influence of different influencing factors on the error rate of the system recognition is identified by comparing the average causal effect size, thereby determining the failure mechanism of the intelligent recognition system.

[0194] [Example] Define the set of nodes in a cause-effect graph. Analyze the causal relationships between nodes and construct a causal graph between influencing factors and the system's error rate, such as... Figure 3 As shown, a causal inference model based on the Do operator intervention is constructed. The model's input consists of a causal graph between influencing factors and system identification error rates, along with simulation test results. The output is the average causal effect of the target influencing factor on system identification error rates. Robustness is tested using random common factor rebuttal. The experimental results are shown in the table below:

[0195] Table 4. Calculation results of average causal effect

[0196]

[0197]

[0198] By comparing the magnitude of the average causal effect, the relative strength of the influence of different factors on the system's recognition error rate is identified. The order of the influence of the factors on the system's recognition error rate is as follows: (Intelligent model architecture type) (Whether the identification sample and the registered sample channel are the same) (Signal-to-noise ratio of the identified sample) (Sample identification time) (Registration sample duration) (Relative duration of the training dataset). This sequence can serve as an important reference for determining the failure mechanism of an intelligent recognition system.

Claims

1. A method for analyzing the failure mechanism of an intelligent identification system based on Do operator intervention, characterized in that... It includes the following steps: Step 1: Construct the task scenarios for the intelligent recognition system and define the system's recognition error rates; The task process of the intelligent recognition system is divided into three stages: model training, feature registration, and identity recognition. The false alarm rate is defined as the probability that a false positive sample is accepted when the system performs the recognition task, and its calculation formula is as follows: Where FP is the number of false positive samples, TN is the number of true negative samples, and FAR is the false alarm rate; The false negative rate is defined as the probability that a true negative sample is rejected, and its calculation formula is as follows: Where FN is the number of false negative samples, TP is the number of true positive samples, and FRR is the false negative rate; The error rate at which the false positive rate equals the false negative rate is defined as the system identification error rate, and its calculation formula is as follows: FAR = FRR = EER Wherein, EER represents the system's error rate in identifying tasks; Step 2: Analyze the key influencing factors in the task scenario, determine the data structure of "influencing factors-evaluation parameters-feature parameters", and construct a factor influencing model of the system's recognition error rate; Step 1: Analyze the key influencing factors in each task scenario; Analyze the key influencing factors in the three task stages of intelligent recognition system: model training, feature registration, and identity recognition, and obtain the set of influencing factors, denoted as . Where i = {1, 2, 3}, m ij Let n represent the j-th influencing factor in the i-th stage. i This represents the number of influencing factors in stage i; Step 2: Construct a data structure of "influencing factors - evaluation parameters - characteristic parameters"; Analyze the various influencing factors obtained in Step 1 of Step 2, select a suitable quantitative evaluation parameter that can characterize the influencing factor and its changing trend, and obtain the set of evaluation parameters for the influencing factor, denoted as . Where i = {1, 2, 3}, p ij Indicates influencing factor m ij Evaluation parameters; Based on the characteristics of parameter values ​​and whether the values ​​are ordered, evaluation parameters are divided into numerical parameters, categorical parameters, and degree parameters; the classification rules for these three types of parameters are as follows: (1) Classification rules for numerical parameters: values ​​are continuous real numbers; (2) Classification rules for categorical parameters: can be quantified into discrete real numbers, and the values ​​are unordered; (3) Classification rules for degree-type parameters: can be quantified into discrete real numbers, and the values ​​are ordered; Analyze each evaluation parameter and select one or more characteristic parameters according to the different parameter categories: (1) For numerical parameters, their values ​​are defined as characteristic parameters, and the number of characteristic parameters is 1; (2) For a categorical parameter p, the number of its feature parameters is determined using a binary encoding method; first, the category of parameter p is encoded, denoted as p Type ={1,2,...,n p }, where p Type n represents the set of category codes for parameter p. p This represents the total number of categories for the parameter; then, the category encoding of the parameter is converted into a binary representation, with each binary bit representing a feature parameter; the rule for determining the number of binary bits based on the total number of parameter categories is as follows: ①n p ≤2: The number of binary bits is 1, and the number of feature parameters is 1; ②2<n p ≤4: The number of binary bits is 2, and the number of feature parameters is 2; ③4<n p ≤8: The number of binary bits is 3, and the number of feature parameters is 3; (3) For degree-type parameters, numerical values ​​are generated by interval mapping. The degree-type parameters are numerically generated according to the parameter value order and relative size, and mapped to the interval (0,1). Based on the above analysis, a set of feature parameters is obtained, denoted as C = {c p |p∈P i ;i=1,2,3}, where c p The characteristic parameters representing the evaluation parameter p; Complete the construction of the "Influencing Factors-Evaluation Parameters-Feature Parameters" data structure for the intelligent recognition system, namely the "MPC" data structure, where M = {M i }, i={1,2,3},P={P i }, i = {1, 2, 3}; Step 3: Construct a model of factors influencing the error rate of system identification, etc.; A factor-driven model is constructed to influence the error rate of the system recognition, simulating the impact of key influencing factors on the recognition task results of the intelligent recognition system. The model is as follows: EER=f(C)+g(U EER ) Where f(C) represents the influence of feature parameter C on the system's recognition error rate, g(U EER ) represents random influencing factor U EER Impact on system recognition and other error rates; Step 3: Create a test dataset containing different defects and obtain test data through simulation experiments; Step 1: Adjust the values ​​of the characteristic parameters of each influencing factor, conduct preliminary experiments, and obtain the range of zero-defect test data. For influencing factor m 11 Its evaluation parameter is p 11 The characteristic parameters are Keeping the values ​​of the characteristic parameters of other influencing factors constant, adjust The value of m is recorded to determine the error rate of the system's recognition task and obtain the influencing factor m. 11 Feature parameter c p11 The range of values ​​that have no impact on the error rate of system recognition, etc., is denoted as (c p11 ,c p11 ), where c p11 Indicates the characteristic parameter c p11 The minimum value, c, that has no impact on the error rate of system recognition, etc. p11 Indicates the characteristic parameter c p11 The maximum value that has no impact on the error rate of system recognition, etc.; Repeat the above process to obtain the set of value ranges of all influencing factors' feature parameters that have no impact on the system's recognition error rate, denoted as R = {(c p ,c p )|p∈P i ;i=1,2,3};Based on this, the range of zero-defect test data values ​​is defined as R0={(c p ,c p )|p∈P i ;i=1,2,3}; Step 2: Create a test dataset; The test dataset consists of three subsets: the intelligent model set, the registration set, and the identification set. The registration set and the identification set each include samples with defects and samples without defects, respectively. (1) Create an intelligent model set: By modifying the feature parameter values ​​of the influencing factors during the model training phase, different intelligent recognition models are obtained; create an intelligent model set, including N1 intelligent models; (2) Create a defective sample set: Modify the values ​​of the relevant feature parameters of each sample data. When the values ​​of q feature parameters are not within the range R0 of the zero-defect test data, it is considered that q defects have been injected into the test data. Create a defective sample set, including: N2 registered samples with defects and N3 identified samples with defects. (3) Create a sample set without defects: Modify the values ​​of the relevant feature parameters of each sample data to ensure that the values ​​of all feature parameters are within the range R0 of zero-defect test data, and are regarded as test data without defects; create a sample set without defects, including: N'2 samples of the registration set without defects and N3' samples of the identification set without defects. (4) Create a test dataset: merge the intelligent model set, the defective sample set and the non-defective sample set to obtain a test dataset, which includes three subsets: intelligent model set (including N1 intelligent models), registration set (including N2 defective registration set samples and N'2 non-defective registration set samples) and identification set (including N3 defective identification set samples and N'3 non-defective identification set samples); Step 3: Obtain test data through simulation experiments; The simulation experiment was conducted using the test dataset obtained in step 2 of step 3. The simulation test steps are as follows: (1) Select an intelligent recognition model from the set of intelligent models; (2) Select M registration samples with the same feature parameter values ​​from the registration set and register them in the selected intelligent recognition model, where M≤N2+N'2, and build an intelligent recognition system. (3) Randomly select O identification samples with the same feature parameter values ​​from the identification set for identification testing, where O≤N3+N3', and record the system identification error rate; obtain 1 test result data, which includes two parts: the feature parameter values ​​of all influencing factors and the system identification error rate; (4) Repeat steps (1) to (3) Z times, and finally obtain Z test result data; Step 4: Process the test data and normalize the evaluation parameter data of each influencing factor; Categorical parameters, after binary encoding, already have values ​​within the range (0,1), requiring no further normalization. Degree parameters are numerically represented and mapped to the range (0,1) using an interval mapping method, also requiring no further normalization. The feature parameters of numerical parameters are normalized using the following formula: Where c' is the normalized data of data c, c max c min These are the maximum and minimum values ​​of data c in the test results data, respectively; The system recognition error rate (EER) ranges from [0, 0.5]. After normalization, its calculation formula is as follows: EER'=2×EER Obtain the test data matrix Where D i This represents the i-th test data. This indicates that each test data point includes normalized data of characteristic parameters of influencing factors in all stages of the intelligent recognition system and the system's recognition error rate; Step 5: Build a causal inference model based on Do operator intervention, estimate the average causal effect, and conduct failure mechanism analysis; Step 1: Construct a causal graph relating influencing factors to the error rate of system identification, etc. One feature parameter corresponds to one node in the causal graph, and the data EER' corresponds to the random influencing factor U. EER Each node in the causal graph corresponds to a node in the causal graph. The set of nodes in the causal graph, V, is defined as denoted as . Where K represents the sum of characteristic parameters of the influencing factors under investigation; based on the factor influence model of system identification error rate, the causal relationship between nodes is analyzed to obtain the causal relationship set E = {(V i V j )}, where (V i V j ) represents V i For V j The reason, V j For V i The results; construct a causal graph G(V,E) between influencing factors and error rates such as system identification; Step 2: Define the average causal effect; The average causal effect is used to quantify the impact of changes in the values ​​of target influencing factors on the error rate of system identification, etc. The calculation formula is as follows: ACE=E(V EER' |Do(V=v))-E(V EER' |Do(V=v')) Where ACE is the average causal effect, E(V) EER' |Do(V=v)) represents intervening in node V such that when the value of V is v, node V... EER' The mean of the values, U EER It is a random factor, and its node mean is 0; According to the Do operator inference rule, the above formula is simplified and transformed into a formula for calculating the causal effect between variables using observational data. The calculation formula is as follows: Among them, V a It is all relatively ordered pairs of nodes (V,V) EER' The set of variables that satisfy the backdoor criterion, i.e., V a Given values ​​for all variables within the set, node V and node V EER' They are independent of each other; Step 3: Construct a causal inference model based on Do operator intervention; A causal inference model based on Do operator intervention is constructed. The input of the model is the causal graph between the influencing factors and the error rate of system identification, etc., and the simulation test data. The output is the average causal effect of the target influencing factor on the error rate of system identification, etc. The model operation process is as follows: Do intervention is performed on the target influencing factor, the causal relationship between the target influencing factor and other factors changes, the causal graph between the target influencing factor and the error rate of system identification, etc. is updated, and the average causal effect is estimated using the Do operator inference rule to obtain the average causal effect of the target influencing factor on the error rate of system identification, etc. Step 4: Conduct a robustness test to decide whether to accept the results; Robustness test is performed using random common factor refutation. Independent random variables are added to the dataset and the causal effect value is estimated again. If the estimated result after refutation is not much different from the original result, the result is accepted and the process proceeds to step 5. Otherwise, the result is rejected and step 1 of step 5 is repeated. Step 5: Analyze the failure mechanism of the intelligent recognition system; Based on the results obtained in step 3, the influence of different influencing factors on the error rate of the system recognition is identified by comparing the average causal effect size, thereby determining the failure mechanism of the intelligent recognition system.

Images