Intelligent identification system failure mechanism analysis method based on Do operator intervention

Abstract

The invention provides an intelligent identification system failure mechanism analysis method based on Do operator intervention for an intelligent identification system. The method comprises the following four steps: (1) constructing a task scene of an intelligent identification system, and defining error rates such as system identification; (2) analyzing key influence factors in a task scene, determining an influence factor-evaluation parameter-characteristic parameter data structure, and constructing a factor influence model of error rates such as system identification; (3) manufacturing a test data set containing different defects, and obtaining test data through a simulation test; (4) processing test data, and performing normalization processing on evaluation parameter data of each influence factor; and (5) building a causal inference model based on Do operator intervention, estimating an average causal effect, and performing failure mechanism analysis.

Description

Technical Field

[0001] The present invention is oriented to intelligent recognition systems, and proposes a failure mechanism analysis method for intelligent recognition systems based on Do operator intervention. The method is aimed at intelligent recognition systems, analyzes key influencing factors that may cause system recognition failure in the entire task process, constructs a factor influence model of system recognition error rate, makes a defective test data set to obtain test data, constructs a causal graph, builds a causal inference model based on Do operator intervention, and uses Do operator reasoning rules to estimate the average causal effect of influencing factors on system recognition failure, so as to better identify the influence of different influencing factors on system recognition results, and provide a reference for the failure mechanism analysis of intelligent recognition systems. The present invention belongs to the field of artificial intelligence. Background Art

[0002] As the accuracy and intelligence level of biometric technology continue to improve, intelligent recognition technology has gradually occupied an important position in the field of biometric identification and has been widely used in finance, criminal investigation, justice and other fields. At the same time, the security and reliability of intelligent recognition systems in the task process have also attracted more attention. A comprehensive and complete system failure mechanism analysis method is needed to evaluate the influence of the influencing factors on system failure and provide a basis for the design optimization of intelligent recognition systems.

[0003] Intelligent recognition systems are often affected by a variety of factors during the execution of recognition tasks. In the intelligent model training stage, the size and language of the training data set, the general model architecture used, and different training strategies will affect the system recognition effect; in the system commissioning stage, the length, noise, and acquisition method of the registration sample and the test sample will also affect the system recognition effect. At present, the relevant research on fault mechanism analysis for intelligent recognition systems is not comprehensive enough, and there is a lack of failure mechanism analysis methods that can cover the entire task stage. In view of this, the present invention proposes a failure mechanism analysis method for intelligent recognition systems based on Do operator intervention. By analyzing the task profile in the recognition task of the intelligent recognition system, comprehensively analyzing the key influencing factors of each task stage, selecting appropriate evaluation parameters for each influencing factor, making a test data set containing different defects, performing simulation tests to obtain test data, using a causal graph model to represent the causal relationship between variables, building a causal inference model based on Do operator intervention, and using Do operator intervention and Do operator inference rules to estimate the average causal effect of influencing factors on system recognition failure, so as to identify the influence of different influencing factors on the system recognition results, and provide a reference for the fault mechanism analysis and design optimization of the intelligent voiceprint recognition system. Summary of the invention

[0004] The present invention provides a method for analyzing failure mechanism of intelligent recognition system based on Do operator intervention, see Figure 1The main steps of the present invention are as follows:

[0005] Step 1: Construct the task scenario of the intelligent recognition system and define the error rate of system recognition;

[0006] The task process of the intelligent recognition system is divided into three stages: model training, feature registration, and identity recognition. The probability of a false positive sample being accepted when the system performs the recognition task is defined as the false alarm rate, and its calculation formula is:

[0007]

[0008] Among them, FP is the number of false positive samples, TN is the number of true negative samples, and FAR is the false alarm rate.

[0009] The probability of a true negative sample being rejected is defined as the false negative rate, and its calculation formula is:

[0010]

[0011] Among them, FN is the number of false negative samples, TP is the number of true positive samples, and FRR is the false negative rate.

[0012] The error rate when the false alarm rate is equal to the missed alarm rate is defined as the system identification error rate, and its calculation formula is:

[0013] FAR=FRR=EER

[0014] Among them, EER represents the equal error rate of the system recognition task.

[0015] Step 2: Analyze the key influencing factors in the task scenario, determine the "influencing factors-evaluation parameters-characteristic parameters" data structure, and build a factor influencing model for system identification and error rate;

[0016] Step 1: Analyze the key influencing factors in each task scenario;

[0017] Analyze the key influencing factors of the three task stages of intelligent recognition system model training, feature registration, and identity recognition, and obtain the influencing factor set, recorded as where i = {1, 2, 3}, m ij represents the jth influencing factor in the i-th stage, n i Represents the number of influencing factors in stage i.

[0018] Step 2: Construct the "influencing factors-evaluation parameters-characteristic parameters" data structure;

[0019] Analyze each influencing factor obtained in step 2 and step 1, select a suitable quantitative evaluation parameter that can characterize the influencing factor and its change trend, and obtain the evaluation parameter set of the influencing factor, which is recorded as where i = {1, 2, 3}, p ij Influencing factors m ij evaluation parameters.

[0020] According to the characteristics of parameter values ​​and whether different parameter values ​​are ordered, the evaluation parameters are divided into numerical parameters, category parameters, and degree parameters. The classification rules for the three types of parameters are as follows:

[0021] (1) Classification rules for numerical parameters: the values ​​are continuous real numbers;

[0022] (2) Classification rules for categorical parameters: they can be numerically converted into discrete real numbers, and there is no order between the values;

[0023] (3) Classification rules for degree parameters: They can be numerically converted into discrete real numbers, and the values ​​are ordered.

[0024] Analyze each evaluation parameter and select one or more characteristic parameters according to different parameter categories:

[0025] (1) For numerical parameters, define their values ​​as characteristic parameters, and the number of characteristic parameters is 1;

[0026] (2) For the categorical parameter p, the binary encoding method is used to determine the number of its characteristic parameters. First, the category of parameter p is encoded, denoted as p Type ={1,2,...,n p}, where p Type Represents the category encoding set of parameter p, n p Indicates the total number of categories of the parameter; then convert the parameter category code into binary representation, and each binary bit is a characteristic parameter. The rules for determining the number of binary bits based on the total number of parameter categories are as follows:

[0027] ①n p ≤2: The number of binary digits is 1 and the number of characteristic parameters is 1;

[0028] ②2<n p ≤4: The number of binary digits is 2, and the number of characteristic parameters is 2;

[0029] ③4<n p ≤8: The number of binary digits is 3 and the number of characteristic parameters is 3.

[0030] (3) For degree parameters, the interval mapping method is used to digitize them. According to the order and relative size of the parameter values, the degree parameters are digitized and mapped to the interval (0,1).

[0031] Based on the above analysis, a characteristic parameter set is obtained, denoted as C = {c p |p∈P i ; i = 1, 2, 3}, where c pRepresents the characteristic parameter of the evaluation parameter p.

[0032] Complete the construction of the data structure of "influencing factors-evaluation parameters-characteristic parameters" of the intelligent recognition system, namely the "MPC" data structure, where M = {M i}, i = {1, 2, 3}, P = {P i},i={1,2,3}.

[0033] Step 3: Construct a model for factors affecting the error rate of system identification;

[0034] Construct a model of factors affecting the system recognition error rate to simulate the impact of key factors on the recognition task results of the intelligent recognition system. The model is:

[0035] EER=f(C)+g(U EER )

[0036] Among them, f(C) represents the influence of characteristic parameter C on the error rate of system recognition, g(U EER ) represents the random influencing factor U EER Impact on system recognition error rate, etc.

[0037] Step 3: Create a test data set containing different defects and obtain test data through simulation experiments;

[0038] Step 1: Adjust the values ​​of the characteristic parameters of each influencing factor, conduct a preliminary experiment, and obtain the value range of zero-defect test data;

[0039] For the influencing factor m 11 , whose evaluation parameter is p 11 , the characteristic parameters are Control the values ​​of the characteristic parameters of other influencing factors to remain unchanged and adjust The value of , record the equal error rate of the system recognition task, and obtain the influencing factor m 11 Characteristic parameters The value range that has no effect on the system recognition error rate is denoted as in Represents characteristic parameters The minimum value that has no effect on the system recognition error rate, Represents the characteristic parameter c p11 The maximum value that has no effect on the system recognition error rate.

[0040] Repeat the above process to obtain the value range set of the characteristic parameters of all influencing factors that have no effect on the system recognition error rate, recorded as Based on this, the value range of zero-defect test data is defined as

[0041] Step 2: Create a test dataset;

[0042] The test data set includes three subsets: intelligent model set, registration set and recognition set, where the registration set and recognition set include samples with defects and samples without defects respectively.

[0043] (1) Create an intelligent model set: By modifying the characteristic parameter values ​​of the factors affecting the model training phase, different intelligent recognition models are trained. Create an intelligent model set, including N 1 An intelligent model.

[0044] (2) Create a defective sample set: Modify the relevant characteristic parameter values ​​of each sample data. When the values ​​of q characteristic parameters are not within the value range R of the zero-defect test data, 0 When q defects are injected into the test data, the defect sample set is created, including: 2 The number of samples of the defective identification set N 3 strip.

[0045] (3) Create a sample set without defects: Modify the relevant characteristic parameter values ​​of each sample data to ensure that the values ​​of all characteristic parameters are within the zero-defect test data value range R 0 The sample set without defects is made, including: the registered sample set without defects N' 2 N, the sample of the recognition set without defects 3 'strip.

[0046] (4) Prepare the test data set: Combine the intelligent model set, the sample set with defects, and the sample set without defects to obtain the test data set, which includes three subsets: the intelligent model set (including N 1 Intelligent models), registration set (including registration set samples with defects N 2 , the number of defect-free registration set samples N' 2 Items) and the identification set (including the identification set sample N containing defects 3 N, the sample of the recognition set without defects 3 'strip).

[0047] Step 3: Obtain test data through simulation experiments;

[0048] Use the test data set obtained in step 2 of step 3 to perform simulation experiments. The simulation test steps are:

[0049] (1) Select an intelligent recognition model from the intelligent model set;

[0050] (2) Select M registered samples with the same feature parameter values ​​from the registration set and register them in the selected intelligent recognition model, where M≤N 2+N' 2 , build an intelligent recognition system;

[0051] (3) Randomly select O recognition samples with the same feature parameter values ​​in the recognition set for recognition test, where O ≤ N 3 +N 3 ', record the error rate of system identification, etc. Obtain 1 test result data, the test result data includes two parts: the characteristic parameter values ​​of all influencing factors and the error rate of system identification, etc.

[0052] (4) Repeat steps (1) to (3) Z times, and finally obtain Z test result data.

[0053] Step 4: Process the test data and normalize the evaluation parameter data of each influencing factor;

[0054] After the categorical parameters are processed by binary encoding, the value range is already between (0,1), and no normalization is required. The degree parameters are digitized and mapped to the interval (0,1) through the interval mapping method, and no normalization is required. The characteristic parameters of the numerical parameters are normalized, and the calculation formula is:

[0055]

[0056] Among them, c' is the data after normalization of data c, c max 、c min are the maximum and minimum values ​​of data c in the test result data respectively.

[0057] The value range of the system recognition error rate EER is [0,0.5]. After normalization, the calculation formula is:

[0058] EER'=2×EER

[0059] Get the test data matrix Where D i represents the i-th test data, It means that each piece of test data includes the normalized data of the characteristic parameters of the influencing factors of the intelligent recognition system in the whole task stage and the error rate of the system recognition.

[0060] Step 5: Construct a causal inference model based on Do operator intervention, estimate the average causal effect, and perform failure mechanism analysis;

[0061] Step 1: Construct a causal diagram between the influencing factors and the error rate of system identification;

[0062] A characteristic parameter corresponds to a node in the causal graph, and the data EER' is related to the random influencing factor U EERThey also correspond to a node in the causal graph respectively, and the node set V of the causal graph is defined as Where K represents the sum of characteristic parameters of the influencing factors to be explored. According to the factor influencing model of system identification error rate, the causal relationship between nodes is analyzed to obtain the causal relationship set E = {(V i ,V j )}, where (V i ,V j ) indicates V i V j The reason, V j V i Construct a causal graph G(V,E) between the influencing factors and the system identification error rate.

[0063] Step 2: Define the average causal effect;

[0064] The average causal effect is used to quantify the impact of the change in the value of the target influencing factor on the value of the system identification error rate. The calculation formula is:

[0065] ACE=E(V EER' |Do(V=v))-E(V EER' |Do(V=v'))

[0066] Among them, ACE is the average causal effect, E(V EER' |Do(V=v)) means to intervene in node V, so that the value of V is v when node V EER' The mean value, U EER is a random factor, and its node mean is 0.

[0067] According to the Do operator inference rule, the above formula is simplified and converted into a causal effect formula that can directly use observational data to calculate the causal effect between variables. The calculation formula is:

[0068]

[0069] Among them, V a are all relatively ordered pairs of nodes (V,V EER' ) is the set of variables that satisfy the backdoor criteria, namely V a After all variables in the set are given values, nodes V and V EER' Independent of each other.

[0070] Step 3: Construct a causal inference model based on Do operator intervention;

[0071] Construct a causal inference model based on Do operator intervention. The input of the model is the causal diagram between the influencing factors and the system identification error rate and the simulation test data; the output is the average causal effect of the target influencing factors on the system identification error rate. The model operation process is: implement the Do intervention operation on the target influencing factors, the causal relationship between the target influencing factors and other factors changes, and by updating the causal diagram between the target influencing factors and the system identification error rate, use the Do operator inference rule to estimate the average causal effect, and obtain the average causal effect of the target influencing factors on the system identification error rate.

[0072] Step 4: Perform a robustness test and decide whether to accept the result;

[0073] Use random common factor refutation to perform robustness test, add independent random variables to the data set and estimate the causal effect value again. If the estimated result after refutation is not much different from the original result, accept the result and proceed to the next step 5; otherwise reject the result and repeat step 5 step 1.

[0074] Step 5: Analyze the failure mechanism of the intelligent recognition system;

[0075] According to the results obtained in step 3, the influence of different influencing factors on the system recognition error rate is identified by comparing the average causal effect size, thereby determining the failure mechanism of the intelligent recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A method flow for analyzing failure mechanism of intelligent identification system based on Do operator intervention

[0077] Figure 2 Causal inference model based on Do operator intervention

[0078] Figure 3 Cause and effect diagram between influencing factors and system identification error rate Specific implementation methods

[0080] Step 1: Construct the task scenario of the intelligent recognition system and define the error rate of system recognition;

[0081] The task process of the intelligent recognition system is divided into three stages: model training, feature registration, and identity recognition. The probability of a false positive sample being accepted when the system performs the recognition task is defined as the false alarm rate, and its calculation formula is:

[0082]

[0083] Among them, FP is the number of false positive samples, TN is the number of true negative samples, and FAR is the false alarm rate.

[0084] The probability of a true negative sample being rejected is defined as the false negative rate, and its calculation formula is:

[0085]

[0086] Among them, FN is the number of false negative samples, TP is the number of true positive samples, and FRR is the false negative rate.

[0087] The error rate when the false alarm rate is equal to the missed alarm rate is defined as the system identification error rate, and its calculation formula is:

[0088] FAR=FRR=EER

[0089] Among them, EER represents the equal error rate of the system recognition task.

[0090] [Example] The task process of the intelligent voiceprint recognition system is divided into three stages: model training, feature registration, and identity recognition. In the identity recognition task, as the set similarity threshold continues to increase, the system recognition false alarm rate FAR continues to decrease, and the false alarm rate FRR continues to increase. Assuming that when the similarity threshold is set to 0.6, the false alarm rate and false alarm rate are equal, both 8%, then the system recognition error rate is:

[0091] EER=FAR=FRR=8%

[0092] Step 2: Analyze the key influencing factors in the task scenario, construct the "influencing factors-evaluation parameters-characteristic parameters" data structure, and build a factor influencing model for the system identification error rate;

[0093] Step 1: Analyze the key influencing factors in each task scenario;

[0094] Analyze the key influencing factors of the three task stages of intelligent recognition system model training, feature registration, and identity recognition, and obtain the influencing factor set, recorded as where i = {1, 2, 3}, m ij represents the jth influencing factor in the i-th stage, n i Represents the number of influencing factors in stage i.

[0095] Step 2: Construct the "influencing factors-evaluation parameters-characteristic parameters" data structure;

[0096] Analyze each influencing factor obtained in step 2 and step 1, select a suitable quantitative evaluation parameter that can characterize the influencing factor and its change trend, and obtain the evaluation parameter set of the influencing factor, which is recorded as where i = {1, 2, 3}, p ij Influencing factors m ij evaluation parameters.

[0097] According to the characteristics of parameter values ​​and whether different parameter values ​​are ordered, the evaluation parameters are divided into numerical parameters, category parameters, and degree parameters. The classification rules for the three types of parameters are as follows:

[0098] (1) Classification rules for numerical parameters: the values ​​are continuous real numbers;

[0099] (2) Classification rules for categorical parameters: they can be numerically converted into discrete real numbers, and there is no order between the values;

[0100] (3) Classification rules for degree parameters: They can be numerically converted into discrete real numbers, and the values ​​are ordered.

[0101] Analyze each evaluation parameter and select one or more characteristic parameters according to different parameter categories:

[0102] (1) For numerical parameters, define their values ​​as characteristic parameters, and the number of characteristic parameters is 1;

[0103] (2) For the categorical parameter p, the binary encoding method is used to determine the number of its characteristic parameters. First, the category of parameter p is encoded, denoted as p Type ={1,2,...,n p}, where p Type Represents the category encoding set of parameter p, n p Indicates the total number of categories of the parameter; then convert the parameter category code into binary representation, and each binary bit is a characteristic parameter. The rules for determining the number of binary bits based on the total number of parameter categories are as follows:

[0104] ①n p ≤2: The number of binary digits is 1 and the number of characteristic parameters is 1;

[0105] ②2<n p ≤4: The number of binary digits is 2, and the number of characteristic parameters is 2;

[0106] ③4<n p ≤8: The number of binary digits is 3 and the number of characteristic parameters is 3.

[0107] (3) For degree parameters, the interval mapping method is used to digitize them. According to the order and relative size of the parameter values, the degree parameters are digitized and mapped to the interval (0,1).

[0108] Based on the above analysis, a characteristic parameter set is obtained, denoted as C = {c p |p∈P i ; i = 1, 2, 3}, where c p Represents the characteristic parameter of the evaluation parameter p.

[0109] Complete the construction of the data structure of "influencing factors-evaluation parameters-characteristic parameters" of the intelligent recognition system, namely the "MPC" data structure, where M = {M i}, i = {1, 2, 3}, P = {P i},i={1,2,3}.

[0110] Step 3: Construct a model for factors affecting the error rate of system identification;

[0111] Construct a model of factors affecting the system recognition error rate to simulate the impact of key factors on the recognition task results of the intelligent recognition system. The model is:

[0112] EER=f(C)+g(U EER )

[0113] Among them, f(C) represents the influence of characteristic parameter C on the error rate of system recognition, g(U EER ) represents the random influencing factor U EER Impact on system recognition error rate, etc.

[0114] [Example] Analyze the key influencing factors of the three task stages of intelligent voiceprint recognition system model training, feature registration, and identity recognition, and obtain the influencing factor set:

[0115] M 1 ={m 11 = training data set size, m 12 =Intelligent Model Architecture}

[0116] M 2 ={m 21 =Registered voice duration}

[0117] M 3 ={m 31 =Recognition speech duration, m 32 =Recognize speech background noise, m 33 =Identify voice channel}

[0118] Among them, the selected training data set is CN-Celeb, and its total duration is set to T; the selected intelligent model architectures include CAM++ model architecture and ECAPA-TDNN model architecture.

[0119] Analyze each influencing factor obtained by the analysis, select appropriate quantitative evaluation parameters that can characterize the influencing factor and its change trend, and obtain the evaluation parameter set of the influencing factor:

[0120] P 1 ={p 11 = Relative length of training data set, p 12 =Intelligent model architecture type}

[0121] P 2 ={p 21 = Registration sample duration}

[0122] P 3 ={p 31 =Recognition sample duration, p 32 = Identification sample signal-to-noise ratio, p 33 =Identify whether the sample channel is the same as the registered sample channel}

[0123] Each evaluation parameter is analyzed, and one or more characteristic parameters are selected according to the parameter category and the specific characteristics of the parameter. The selection results of characteristic parameters of the intelligent voiceprint recognition system are shown in Table 1.

[0124] Table 1. Results of feature parameter selection for intelligent voiceprint recognition system

[0125]

[0126]

[0127] Get the characteristic parameter set C = {c p |p∈P i ; i = 1, 2, 3}, complete the construction of the "influencing factors-evaluation parameters-characteristic parameters" data structure of the intelligent recognition system, that is, the "MPC" data structure.

[0128] Construct a model of factors affecting the system recognition error rate to simulate the impact of key factors on the recognition task results of the intelligent recognition system. The model is:

[0129] EER=f(C)+g(U EER )

[0130] Step 3: Create a test data set containing different defects and obtain test data through simulation experiments;

[0131] Step 1: Adjust the values ​​of the characteristic parameters of each influencing factor, conduct a preliminary experiment, and obtain the value range of zero-defect test data;

[0132] For the influencing factor m 11 , whose evaluation parameter is p 11 , the characteristic parameters are Control the values ​​of the characteristic parameters of other influencing factors to remain unchanged and adjust The value of , record the equal error rate of the system recognition task, and obtain the influencing factor m 11 Characteristic parameters The value range that has no effect on the system recognition error rate is denoted as in Represents characteristic parameters The minimum value that has no effect on the system recognition error rate, Represents characteristic parameters The maximum value that has no effect on the system recognition error rate.

[0133] Repeat the above process to obtain the value range set of the characteristic parameters of all influencing factors that have no effect on the system recognition error rate, recorded as Based on this, the value range of zero-defect test data is defined as

[0134] Step 2: Create a test dataset;

[0135] The test data set includes three subsets: intelligent model set, registration set and recognition set, where the registration set and recognition set include samples with defects and samples without defects respectively.

[0136] (1) Create an intelligent model set: By modifying the characteristic parameter values ​​of the factors affecting the model training phase, different intelligent recognition models are trained. Create an intelligent model set, including N 1 An intelligent model.

[0137] (2) Create a defective sample set: Modify the relevant characteristic parameter values ​​of each sample data. When the values ​​of q characteristic parameters are not within the value range R of the zero-defect test data, 0 When q defects are injected into the test data, the defect sample set is created, including: 2 The number of samples of the defective identification set N 3 strip.

[0138] (3) Create a sample set without defects: Modify the relevant characteristic parameter values ​​of each sample data to ensure that the values ​​of all characteristic parameters are within the zero-defect test data value range R 0 The sample set without defects is made, including: the registered sample set without defects N' 2 N, the sample of the recognition set without defects 3 'strip.

[0139] (4) Prepare the test data set: Combine the intelligent model set, the sample set with defects, and the sample set without defects to obtain the test data set, which includes three subsets: the intelligent model set (including N 1 Intelligent models), registration set (including registration set samples with defects N 2 , the number of defect-free registration set samples N' 2 Items) and the identification set (including the identification set sample N containing defects 3 N, the sample of the recognition set without defects 3 'strip).

[0140] Step 3: Obtain test data through simulation experiments;

[0141] Use the test data set obtained in step 2 of step 3 to perform simulation experiments. The simulation test steps are:

[0142] (1) Select an intelligent recognition model from the intelligent model set;

[0143] (2) Select M registered samples with the same feature parameter values ​​from the registration set and register them in the selected intelligent recognition model, where M≤N 2 +N' 2 , build an intelligent recognition system;

[0144] (3) Randomly select O recognition samples with the same feature parameter values ​​in the recognition set for recognition test, where O ≤ N 3 +N 3 ', record the error rate of system identification, etc. Obtain 1 test result data, the test result data includes two parts: the characteristic parameter values ​​of all influencing factors and the error rate of system identification, etc.

[0145] (4) Repeat steps (1) to (3) Z times, and finally obtain Z test result data.

[0146] [Example] Adjust the values ​​of the characteristic parameters of each influencing factor, conduct a preliminary experiment, and obtain the value range of zero-defect test data:

[0147]

[0148] A test data set was created, including three subsets: an intelligent model set (including 4 intelligent models), a registration set (including 40 registration set samples with defects and 320 registration set samples without defects) and a recognition set (including 12,000 recognition set samples with defects and 2,400 recognition set samples without defects).

[0149] Use the obtained test data set to conduct simulation experiments. The simulation test steps are:

[0150] (1) Select an intelligent recognition model from the intelligent model set;

[0151] (2) Select 10 registered samples with the same feature parameter values ​​from the registration set and register them in the selected intelligent recognition model to build an intelligent recognition system;

[0152] (3) Select 400 identification samples with the same characteristic parameter values ​​in the identification set for identification test, and record the system identification error rate. Obtain 1 test result data, which includes two parts: the characteristic parameter values ​​of all influencing factors and the system identification error rate.

[0153] (4) Repeat steps (1) to (3) 144 times, and finally obtain 144 test result data.

[0154] Some test results are shown in the following table:

[0155] Table 2 Test result data (partial)

[0156]

[0157] Step 4: Process the test data and normalize the evaluation parameter data of each influencing factor;

[0158] After the categorical parameters are processed by binary encoding, the value range is already between (0,1), and no normalization is required. The degree parameters are digitized and mapped to the interval (0,1) through the interval mapping method, and no normalization is required. The characteristic parameters of the numerical parameters are normalized, and the calculation formula is:

[0159]

[0160] Among them, c' is the data after normalization of data c, c max 、c min are the maximum and minimum values ​​of data c in the test result data respectively.

[0161] The value range of the system recognition error rate EER is [0,0.5]. After normalization, the calculation formula is:

[0162] EER'=2×EER

[0163] Get the test data matrix Where D i represents the i-th test data, It means that each piece of test data includes the normalized data of the characteristic parameters of the influencing factors of the intelligent recognition system in the whole task stage and the error rate of the system recognition.

[0164] [Example] Normalize the characteristic parameters of numerical parameters:

[0165] (1) Characteristic parameters (Relative length of training data set) Normalization:

[0166]

[0167] (2) Characteristic parameters (Registration sample duration) Normalization:

[0168]

[0169] (3) Characteristic parameters (Recognition sample duration) Normalization:

[0170]

[0171] (4) Characteristic parameters (Recognition sample signal-to-noise ratio) Normalization:

[0172]

[0173] (5) Normalization of system recognition error rate (EER):

[0174] EER'=2×EER

[0175] The test results after partial normalization are shown in the following table:

[0176] Table 3 Normalized test result data (partial)

[0177]

[0178]

[0179] Step 5: Build a causal inference model based on Do operator intervention, estimate the average causal effect, and perform failure mechanism analysis;

[0180] Step 1: Construct a causal diagram between the influencing factors and the error rate of system identification;

[0181] A characteristic parameter corresponds to a node in the causal graph, and the data EER' is related to the random influencing factor U EER They also correspond to a node in the causal graph respectively, and the node set V of the causal graph is defined as Where K represents the sum of characteristic parameters of the influencing factors to be explored. According to the factor influencing model of system identification error rate, the causal relationship between nodes is analyzed to obtain the causal relationship set E = {(V i ,V j )}, where (V i ,V j ) indicates V i V j The reason, V j V i Construct a causal graph G(V,E) between the influencing factors and the system identification error rate.

[0182] Step 2: Define the average causal effect;

[0183] The average causal effect is used to quantify the impact of the change in the value of the target influencing factor on the value of the system identification error rate. The calculation formula is:

[0184] ACE=E(VEER' |Do(V=v))-E(V EER' |Do(V=v'))

[0185] Among them, ACE is the average causal effect, E(V EER' |Do(V=v)) means to intervene in node V, so that the value of V is v when node V EER' The mean value, U EER is a random factor, and its node mean is 0.

[0186] According to the Do operator inference rule, the above formula is simplified and converted into a causal effect formula that can directly use observational data to calculate the causal effect between variables. The calculation formula is:

[0187]

[0188] Among them, V a are all relatively ordered pairs of nodes (V,V EER' ) is the set of variables that satisfy the backdoor criteria, namely V a After all variables in the set are given values, nodes V and V EER' Independent of each other.

[0189] Step 3: Construct a causal inference model based on Do operator intervention;

[0190] Construct a causal inference model based on Do operator intervention, such as Figure 2 As shown. The input of the model is the causal diagram between the influencing factors and the system identification error rate and the simulation test data; the output is the average causal effect of the target influencing factors on the system identification error rate. The model operation process is: implement the Do intervention operation on the target influencing factors, the causal relationship between the target influencing factors and other factors changes, and by updating the causal diagram between the target influencing factors and the system identification error rate, use the Do operator inference rule to estimate the average causal effect, and obtain the average causal effect of the target influencing factors on the system identification error rate.

[0191] Step 4: Perform a robustness test and decide whether to accept the result;

[0192] Use random common factor refutation to perform robustness test, add independent random variables to the data set and estimate the causal effect value again. If the estimated result after refutation is not much different from the original result, accept the result and proceed to the next step 5; otherwise reject the result and repeat step 5 step 1.

[0193] Step 5: Analyze the failure mechanism of the intelligent recognition system;

[0194] According to the results obtained in step 3, the influence of different influencing factors on the system recognition error rate is identified by comparing the average causal effect size, thereby determining the failure mechanism of the intelligent recognition system.

[0195] [Example] Defining a node set of a causal graph Analyze the causal relationship between nodes and construct a causal graph between influencing factors and system identification error rates, such as Figure 3 As shown. A causal inference model based on Do operator intervention is constructed. The input of the model is the causal diagram between the influencing factors and the system identification error rate and the simulation test result data; the output is the average causal effect of the target influencing factors on the system identification error rate, and a random common factor refutation is used for robustness testing. The test results are shown in the following table:

[0196] Table 4 Calculation results of average causal effect

[0197]

[0198]

[0199] By comparing the average causal effect size, we can identify the influence of different influencing factors on the system identification error rate. The order of influence of influencing factors on the system identification error rate is as follows: (intelligent model architecture type), (Identify whether the sample channel is the same as the registered sample channel). (identification sample signal-to-noise ratio), (Recognition sample duration), (Registration sample duration), (Relative length of training data set). This sequence can be used as an important reference to determine the failure mechanism of intelligent recognition system.

Claims

1. A failure mechanism analysis method for intelligent identification system based on Do operator intervention, characterized in that It contains the following steps: Step 1: Construct the task scenario of the intelligent recognition system and define the error rate of system recognition; The task process of the intelligent recognition system is divided into three stages: model training, feature registration, and identity recognition. The probability of a false positive sample being accepted when the system performs the recognition task is defined as the false alarm rate, and its calculation formula is: Among them, FP is the number of false positive samples, TN is the number of true negative samples, and FAR is the false alarm rate; The probability of a true negative sample being rejected is defined as the false negative rate, and its calculation formula is: Among them, FN is the number of false negative samples, TP is the number of true positive samples, and FRR is the false negative rate; The error rate when the false alarm rate is equal to the missed alarm rate is defined as the system identification error rate, and its calculation formula is: FAR=FRR=EER Among them, EER represents the equal error rate of the system recognition task; Step 2: Analyze the key influencing factors in the task scenario, determine the "influencing factors-evaluation parameters-characteristic parameters" data structure, and build a factor influencing model for system identification and error rate; Step 1: Analyze the key influencing factors in each task scenario; Analyze the key influencing factors of the three task stages of intelligent recognition system model training, feature registration, and identity recognition, and obtain the influencing factor set, recorded as where i = {1, 2, 3}, m ij represents the jth influencing factor in the i-th stage, n i represents the number of influencing factors in stage i; Step 2: Construct the "influencing factors-evaluation parameters-characteristic parameters" data structure; Analyze each influencing factor obtained in step 2 and step 1, select a suitable quantitative evaluation parameter that can characterize the influencing factor and its change trend, and obtain the evaluation parameter set of the influencing factor, which is recorded as where i = {1, 2, 3}, p ij Indicates the influencing factor m ij Evaluation parameters of According to the characteristics of parameter values ​​and whether different parameter values ​​are ordered, the evaluation parameters are divided into numerical parameters, category parameters, and degree parameters; the classification rules of the three types of parameters are as follows: (1) Classification rules for numerical parameters: the values ​​are continuous real numbers; (2) Classification rules for categorical parameters: they can be numerically converted into discrete real numbers, and there is no order between the values; (3) Classification rules for degree parameters: they can be numerically converted into discrete real numbers, and the values ​​are orderly; Analyze each evaluation parameter and select one or more characteristic parameters according to different parameter categories: (1) For numerical parameters, define their values ​​as characteristic parameters, and the number of characteristic parameters is 1; (2) For the categorical parameter p, the binary encoding method is used to determine the number of its characteristic parameters; first, the category of the parameter p is encoded, denoted as p Type ={1,2,...,n p }, where p Type Represents the category encoding set of parameter p, n p Indicates the total number of categories of the parameter; then convert the parameter category code into binary representation, each binary bit is a characteristic parameter; the rules for determining the number of binary bits based on the total number of parameter categories are as follows: ①n p ≤2: The number of binary digits is 1 and the number of characteristic parameters is 1; ②2<n p ≤4: The number of binary digits is 2, and the number of characteristic parameters is 2; ③4<n p ≤8: The number of binary digits is 3 and the number of characteristic parameters is 3; (3) For degree parameters, the interval mapping method is used to digitize them; the degree parameters are digitized according to the order and relative size of the parameter values ​​and mapped to the interval (0,1); Based on the above analysis, a characteristic parameter set is obtained, denoted as C = {c p |p∈P i ; i = 1, 2, 3}, where c p Characteristic parameter representing the evaluation parameter p; Complete the construction of the "influencing factors-evaluation parameters-characteristic parameters" data structure of the intelligent recognition system, namely the "MPC" data structure, where M = {M i }, i = {1, 2, 3}, P = {P i },i={1,2,3}; Step 3: Construct a model for factors affecting the error rate of system identification; Construct a model of factors affecting the system recognition error rate to simulate the impact of key factors on the recognition task results of the intelligent recognition system. The model is: EER=f(C)+g(U EER ) Among them, f(C) represents the influence of characteristic parameter C on the error rate of system recognition, g(U EER ) represents the random influencing factor U EER Impact on system recognition error rate; Step 3: Create a test data set containing different defects and obtain test data through simulation experiments; Step 1: Adjust the values ​​of the characteristic parameters of each influencing factor, conduct a preliminary experiment, and obtain the value range of zero-defect test data; For the influencing factor m 11 , whose evaluation parameter is p 11 , the characteristic parameters are Control the values ​​of the characteristic parameters of other influencing factors unchanged and adjust The value of , record the equal error rate of the system recognition task, and obtain the influencing factor m 11 The characteristic parameter c p11 The value range that has no effect on the system recognition error rate is denoted as (c p11 ,c p11 ), where c p11 Represents the characteristic parameter c p11 The minimum value that has no effect on the system recognition error rate, c p11 Represents the characteristic parameter c p11 The maximum value that has no effect on the system recognition error rate; Repeat the above process to obtain the value range set of the characteristic parameters of all influencing factors that have no effect on the system recognition error rate, denoted as R = {(c p ,c p )|p∈P i ; i = 1, 2, 3}; Based on this, the zero-defect test data value range is defined as R0 = {(c p ,c p )|p∈P i ; i = 1, 2, 3}; Step 2: Create a test dataset; The test data set includes three subsets: intelligent model set, registration set and recognition set, where the registration set and recognition set include samples with defects and samples without defects respectively; (1) Creating an intelligent model set: By modifying the characteristic parameter values ​​of the factors affecting the model training phase, different intelligent recognition models are trained and obtained; creating an intelligent model set, including N1 intelligent models; (2) Create a defective sample set: modify the value of the relevant characteristic parameters of each sample data. When the values ​​of q characteristic parameters are not within the value range R0 of the zero-defect test data, it is considered that q defects are injected into the test data; create a defective sample set, including: N2 defective registration set samples and N3 defective identification set samples; (3) Create a sample set without defects: modify the values ​​of the relevant characteristic parameters of each sample data to ensure that the values ​​of all characteristic parameters are within the value range R0 of the zero-defect test data, and regard them as test data without defects; create a sample set without defects, including: N'2 samples of the registration set without defects and N3' samples of the identification set without defects; (4) Producing a test data set: Merge the intelligent model set, the sample set with defects, and the sample set without defects to obtain a test data set, which includes three subsets: the intelligent model set (including N1 intelligent models), the registration set (including N2 registration set samples with defects and N'2 registration set samples without defects), and the recognition set (including N3 recognition set samples with defects and N'3 recognition set samples without defects); Step 3: Obtain test data through simulation experiments; Use the test data set obtained in step 3, step 2 to perform simulation experiments; the simulation test steps are: (1) Select an intelligent recognition model from the intelligent model set; (2) Select M registration samples with the same feature parameter values ​​from the registration set and register them in the selected intelligent recognition model, where M≤N2+N'2, to build an intelligent recognition system; (3) Randomly select O recognition samples with the same characteristic parameter values ​​in the recognition set for recognition test, where O≤N3+N3', and record the system recognition error rate; obtain 1 test result data, which includes two parts: the characteristic parameter values ​​of all influencing factors and the system recognition error rate; (4) Repeat steps (1) to (3) Z times, and finally obtain Z test result data; Step 4: Process the test data and normalize the evaluation parameter data of each influencing factor; After binary coding, the value range of categorical parameters is already between (0, 1), and no normalization is required. Degree parameters are numerated and mapped to the interval (0, 1) through interval mapping method, and no normalization is required. The characteristic parameters of numerical parameters are normalized, and the calculation formula is: Among them, c' is the data after normalization of data c, c max 、c min are the maximum and minimum values ​​of data c in the test result data respectively; The value range of the system recognition error rate EER is [0,0.5]. After normalization, the calculation formula is: EER'=2×EER Get the test data matrix Where D i represents the i-th test data, Indicates that each test data includes the normalized data of the characteristic parameters of the influencing factors of the intelligent recognition system in the full task stage and the error rate of system recognition; Step 5: Build a causal inference model based on Do operator intervention, estimate the average causal effect, and perform failure mechanism analysis; Step 1: Construct a causal diagram between the influencing factors and the error rate of system identification; A characteristic parameter corresponds to a node in the causal graph, and the data EER' is related to the random influencing factor U EER They also correspond to a node in the causal graph respectively, and the node set V of the causal graph is defined as Where K represents the sum of characteristic parameters of the influencing factors to be explored; according to the factor influence model of system identification error rate, the causal relationship between nodes is analyzed to obtain the causal relationship set E = {(V i ,V j )}, where (V i ,V j ) indicates V i V j The reason, V j V i The result; construct a causal graph G(V,E) between the influencing factors and the system identification error rate; Step 2: Define the average causal effect; The average causal effect is used to quantify the impact of the change in the value of the target influencing factor on the value of the system identification error rate. The calculation formula is: ACE=E(V EER' |Do(V=v))-E(V EER' |Do(V=v')) Among them, ACE is the average causal effect, E(V EER' |Do(V=v)) means to intervene in node V, so that the value of V is v when node V EER' The mean value, U EER is a random factor, and its node mean is 0; According to the Do operator inference rule, the above formula is simplified and converted into a causal effect formula that can directly use observational data to calculate the causal effect between variables. The calculation formula is: Among them, V a are all relatively ordered pairs of nodes (V,V EER' ) is the set of variables that satisfy the backdoor criteria, namely V a After all variables in the set are given values, nodes V and V EER' Independent of each other; Step 3: Construct a causal inference model based on Do operator intervention; A causal inference model based on Do operator intervention is constructed; the input of the model is the causal diagram between the influencing factors and the system identification error rate and the simulation test data; the output is the average causal effect of the target influencing factors on the system identification error rate; the model operation process is: implement Do intervention operations on the target influencing factors, the causal relationship between the target influencing factors and other factors changes, and by updating the causal diagram between the target influencing factors and the system identification error rate, use the Do operator inference rule to estimate the average causal effect, and obtain the average causal effect of the target influencing factors on the system identification error rate; Step 4: Perform a robustness test and decide whether to accept the result; Use random common factor refutation to perform robustness test, add independent random variables to the data set and estimate the causal effect value again. If the estimated result after refutation is not much different from the original result, accept the result and proceed to the next step 5; otherwise reject the result and repeat step 5 step 1; Step 5: Analyze the failure mechanism of the intelligent recognition system; According to the results obtained in step 3, the influence of different influencing factors on the system recognition error rate is identified by comparing the average causal effect size, thereby determining the failure mechanism of the intelligent recognition system.

Images