Intelligent system reasoning evolution failure mechanism analysis method based on defect sample generation
Through the method based on defect sample generation, the inference evolution process of intelligent system is analyzed, the diversity and adversarial defect data samples are generated, and the failure mechanism model is constructed, which solves the problem of difficult to characterize the failure mechanism of intelligent system and realizes the deep-level failure analysis of intelligent systems and design improvements.
Patent Information
- Application Number
- CN202411001423.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-25
AI Technical Summary
The failure mechanism of intelligent systems in the inference evolution stage is difficult to be characterized in depth, and existing methods cannot effectively analyze the failure process of intelligent models, which limits the high-quality development of intelligent systems.
Using a method based on defect sample generation, the key influencing factors are determined by analyzing the inference evolution process of intelligent systems, and the diversity and adversarial defect data samples are generated. The failure mechanism model of the intelligent system is constructed in combination with the agent model, and the output accuracy and influencing factors are analyzed. The relationship between.
It realizes in-depth analysis of the failure mechanism of the intelligent system, provides a theoretical basis for the improvement of intelligent system design, and improves the reliability and recognition accuracy of the intelligent system.
Smart Images

Figure CN118965732A_ABST
Abstract
Description
Technical Field
[0001] The present invention is oriented to intelligent systems and proposes a failure mechanism analysis method for intelligent systems based on defect sample generation technology. This method models the reasoning evolution process of intelligent systems for typical tasks such as intelligent perception, cognition and decision-making, determines the failure criteria and key influencing factors of intelligent models; generates data samples through appropriate defect sample generation methods, and inputs them into the intelligent system for testing to obtain failure data; associates failure data with key influencing factor metrics, and obtains a correlation model between the output accuracy of the intelligent system and its key influencing factors based on a proxy model, thereby analyzing the failure mechanism of the intelligent system. The present invention belongs to the field of artificial intelligence and reliability engineering. Background Art
[0002] In recent years, artificial intelligence technology has been developing continuously, and more and more intelligent systems with neural networks as the core algorithm have been widely used in the fields of autonomous driving, security monitoring, agriculture, face recognition, etc. However, due to the unique problems of unclear failure sources and complex failure links in the use of intelligent systems, it is necessary to propose an adaptive failure mechanism analysis method based on the technical characteristics of intelligent systems to provide a basis for the reliability design of intelligent perception systems.
[0003] When intelligent systems perform reasoning tasks such as target detection and recognition, they are often affected by multiple factors and adversarial attacks. To explore the impact of interference, most existing methods analyze the output results of the software system through a large number of tests, without providing a method for analyzing the failure mechanism of the system and building a model. At the same time, the existing general software system failure mechanism analysis method only provides a qualitative description of the software failure mode and failure cause, and cannot deeply characterize the failure process of the model. The current lack of failure mechanism analysis methods for intelligent systems limits the high-quality development of intelligent systems.
[0004] On this basis, the present invention aims at the problem that the failure mechanism of intelligent systems is difficult to characterize in the reasoning evolution stage, and proposes a failure mechanism analysis method for intelligent systems based on defect sample generation. It deeply analyzes the failure causes of intelligent models and provides a theoretical basis for the design improvement of intelligent systems. Summary of the invention
[0005] Aiming at the problem that the failure mechanism of intelligent system is difficult to characterize in the reasoning evolution stage, the present invention proposes a failure mechanism analysis method of intelligent system reasoning evolution based on defect sample generation. The overall research idea of the invention is as follows: Figure 1 As shown, it is divided into the following steps:
[0006] Step 1: Analyze the reasoning evolution process of the intelligent system, find out its key influencing factors, and define its failure criteria.
[0007] Step 1: Identify the key factors affecting the reliability of the intelligent system:
[0008] In the reasoning evolution stage, the key factors affecting the reliability of the intelligent system are the diversity and adversarial nature of the samples, denoted as Div and Adv respectively. By generating diverse and adversarial defect data samples and inputting them into the intelligent system for testing, the failure data of the intelligent system can be obtained, such as Figure 2 As shown;
[0009] Step 2: Define the failure criteria of the intelligent system:
[0010] The recognition accuracy of the intelligent system is used to define its failure criterion. The recognition accuracy of the intelligent system in the original test set is used as the benchmark, denoted as ACC s , when the recognition accuracy of the intelligent system in other defect sample data is lower than this standard, it is considered that the intelligent system has failed and the failure data needs to be recorded. The specific calculation formula for the recognition accuracy of the intelligent system is:
[0011]
[0012] Among them, ACC represents the recognition accuracy of the intelligent system in the test data set; TP represents the number of positive samples predicted as positive by the intelligent system; FP represents the number of negative samples predicted as positive by the intelligent system, FN represents the number of positive samples predicted as negative by the intelligent system, and TN represents the number of negative samples predicted as negative by the intelligent system. Based on this, the failure criterion of the intelligent system is defined as:
[0013]
[0014] Among them, ACC T Represents the recognition accuracy of the intelligent system in the Tth defect data sample.
[0015] Step 2: Combine the key influencing factors obtained to generate defect data samples.
[0016] Step 1: Generate diverse defect samples based on the diffusion model:
[0017] 1) Sample diversity metrics
[0018] Combined with the Cosin similarity index to measure sample diversity, the generated samples are represented as a vector Sam, and the diversity of the generated samples is represented by calculating the cosine value between the vectors. The calculation formula is:
[0019]
[0020] Div=1-|Cosin|
[0021] Among them, Cosin is the sample similarity index, Sam1 represents the vector of the original sample, Sam2 represents the vector of the generated defect sample, and Div represents the calculated diversity.
[0022] 2) Generation of diverse defect samples based on diffusion model
[0023] Diversity defect samples are generated using the diffusion model. In the forward phase of the diffusion model, noise is gradually added to the N image samples in the original image data set until the original image samples are completely destroyed into Gaussian noise. The change process is expressed as:
[0024]
[0025] Among them, x 1:T represents the state sequence from time step t = 1 to time T, which is a random variable sequence that represents the state evolution of the original sample within a certain time range; x0 represents the state of the original sample at t = 0; x t represents the state of the sample at time t; x t-1 represents the state of the sample at time t-1; and q(x t |x t-1 )for:
[0026]
[0027] Among them, β t is the variance hyperparameter of the Gaussian distribution, satisfying β t ∈(0,1), represents Gaussian distribution, where a = x t is the value of Gaussian distribution, represents the mean, c = β t I represents the current time step x t The covariance matrix of the conditional Gaussian distribution of .
[0028] After obtaining the completely destroyed samples, entropy reduction is performed to gradually convert the noise data into new image data samples. The change process is expressed as:
[0029]
[0030] Among them, p(x 0:T ) represents the joint probability density distribution of the state sequence from t = 0 to T, p(x T ) represents the state probability distribution at time T, that is, the image state after being completely destroyed by Gaussian noise; and p(x t-1 |x t )for:
[0031]
[0032] Among them, μ(x t ,t) represents the mean, which is about x t and time t, ∑(x t ,t) represents the covariance matrix. The same number of defect data samples as the original data set are generated by the diffusion model, and the diversity of the samples generated by the diffusion model is calculated by combining the sample diversity metric Div. The diffusion model is used to generate defect data samples with different Div values, which are input into the intelligent system to test the recognition accuracy of the intelligent system in samples with different diversity values. Combined with the failure criteria defined in step 1, sub-step 2, the failure data is recorded.
[0033] Step 2: Generation of adversarial defect samples based on the Fast Gradient Sign Method (FGSM):
[0034] 1) Sample adversarial metrics
[0035] The L2 norm is used to calculate the adversarial nature of the sample. The generated adversarial sample and the original sample are represented as a vector. The adversarial nature calculation formula is expressed as:
[0036]
[0037] Among them, x represents the pixel vector of the original image, y represents the pixel vector of the generated adversarial image, and x i and i are the pixel vectors of the two images at the i-th position, n represents the total number of pixels in the image, and the calculated Adv is the L2 norm.
[0038] 2) Generation of adversarial defect samples based on FGSM
[0039] The calculation formula of the adversarial sample generation process based on FGSM is:
[0040]
[0041] Among them, ε is the perturbation amplitude or perturbation size, sigm(·) is the sign function, using the sign of the gradient, J represents the loss function, θ represents the model parameters, and lab is the label corresponding to the image x. represents the gradient of the loss function J with respect to the input x, and y represents the generated adversarial defect sample. While using FGSM, image rotation and flipping transformations are also added to enhance the adversarial nature of the sample:
[0042]
[0043] Among them, y is the adversarial sample after adding disturbance, p is the flip probability of the adversarial sample, q is the rotation probability of the adversarial sample, TF(y) is the flipped sample, and TL(y) is the rotated sample. Use the adversarial metric to calculate the adversarial value Adv of the generated adversarial sample. By generating adversarial defect data samples with different Adv and inputting them into the intelligent system, the recognition accuracy of the intelligent system in different adversarial defect data samples is tested. Combined with the failure criteria defined in step 1, sub-step 2, the failure data is recorded.
[0044] Step 3: Generation of defect samples with diversity and adversarial coupling:
[0045] 1) Generation of diverse defect samples
[0046] Generate diverse defect samples based on the diffusion model of step 2, sub-step 1, and calculate the diversity Div of the samples through the sample diversity metric;
[0047] 2) Increase the adversarial nature of samples
[0048] Based on the diversity samples, combined with the adversarial defect sample generation method proposed in step 2, we generate diverse and adversarially coupled test samples, and calculate the adversarial Adv of the samples in combination with the sample adversarial metrics.
[0049] The generated diverse and adversarially coupled defect samples are input into the intelligent system to test the recognition accuracy of the intelligent system in the coupled defect data samples. Combined with the failure criteria defined in sub-step 2 of step 1, the failure data is recorded.
[0050] Step 3: Visual failure mechanism analysis based on Grad-CAM.
[0051] Through Grad-CAM, back propagation is performed from the output of the intelligent system to obtain the gradient information of the feature layer before the fully connected layer in the intelligent system, and the average gradient of each feature map at this level is obtained. The expression is:
[0052]
[0053] In the formula, is the average gradient of the k-th channel feature map in the c-th output back propagation; Z is the total number of pixels in the feature image; y c is the maximum class output value output by the intelligent system; A represents the activation matrix, i and j are the row index and column index in the activation matrix respectively, k represents the channel index, is the feature map corresponding to the kth channel. The average gradient of each channel feature map is used as the weight to multiply each channel feature map and superimpose them, and then the negative values less than 0 are removed to eliminate the influence of other categories, and the key area used by the network for classification is obtained. Its expression is:
[0054]
[0055] In the formula, represents the class activation matrix obtained by Grad-CAM; ReLU is a nonlinear activation function used to remove negative values. Multiplying with the original input image and performing weighted summation, we can get the target recognition result visualized by the intelligent system, which is expressed as:
[0056]
[0057] Where X is the input image data passed through the preprocessing layer of the neural network. From the images generated by Grad-CAM, we can observe the changes in the focus area as the diversity and adversarial nature of the test samples change, such as Figure 5 As shown, the reasons for the intelligent system's misidentification are explained.
[0058] Step 4: Construct the failure mechanism model of the intelligent system based on the agent model.
[0059] Step 1: Construction of single factor failure mechanism model of intelligent system based on agent model:
[0060] Arrange the failure data recorded in step 2, sub-step 1 and sub-step 2, in order of diversity and multi-resistance values from small to large, and record them as:
[0061] ((Div1,ACC1),(Div2,ACC2),…,(Div m , ACC num_1 ))
[0062] ((Adv1, ACC1), (Adv2, ACC2), ..., (Adv m , ACC num_2 ))
[0063] Among them, (Div i ,ACC i )(i∈(1,2,...,num_1)) represents the sample diversity value and the recognition accuracy of the intelligent system under the diversity sample, num_1 represents the number of defect data sets with different diversities, (Adv i ,ACC i)(i∈(1,2,...,num_2)) represents the sample adversarial index and the corresponding recognition accuracy of the intelligent system under the adversarial sample, and num_2 represents the number of defect data sets with different adversarial characteristics. Using the failure data, a proxy model based on polynomial fitting is constructed to obtain a single factor mechanism model of the intelligent system, which is expressed as
[0064]
[0065] Among them, in is the sample diversity Div or sample adversarial Adv; represents a single factor mechanism model, which is a function of in; n represents the number of failure data; a i (i∈(1,2,...,n)) represents the parameters in the polynomial. The least squares method is used to solve the model, which is:
[0066]
[0067] Among them, δ(a0,…,a n ) represents the sum of the differences between the model and the failure data, f i represents the actual value of the i-th failure data point. Solve for a0, a1, …, a n The value of makes the value of δ minimum. Taking the derivative of both sides of the function, we get:
[0068]
[0069] ......
[0071]
[0072] This gives the normal equations:
[0073]
[0074] Solve the system of equations and calculate a0,a1,…,a n The value of , the single factor failure mechanism model of the intelligent system is obtained
[0075] Step 2: Construction of multi-factor failure mechanism model of intelligent system based on neural network
[0076] Arrange the failure data recorded in step 3 of step 2 in ascending order of diversity value, and record it as:
[0077] ((Div1, Adv1, ACC1), (Div2, Adv2, ACC2), ..., (Div z , Adv z , ACCz )
[0078] Where z is the number of failure data. Based on the failure data obtained from the test and combined with the deep neural network, a failure mechanism model of the intelligent system under multi-factor coupling is constructed, which is expressed as:
[0079] Φ=g((Div i ,Adv i ,ACC i );θ)+ε i
[0080] Among them, g(·) represents the deep neural network model, (Div i ,Adv i ,ACC i ) is the failure data of the intelligent system, θ represents the neural network parameters, ε i is the error term, and Φ represents the multi-factor mechanism model. Using the failure data of the intelligent system, the network parameters are optimized by minimizing the loss function, which is expressed as:
[0081]
[0082] Among them, L min represents the loss function, ACC j Indicates that the diversity and adversarial values are Div j ,Adv j The intelligent system recognition accuracy corresponding to the j-th failure data, Φ j Indicates that the sample diversity and diversity values are Div j ,Adv j The recognition accuracy of the intelligent system predicted by the multi-factor mechanism model is obtained. After training, the optimal network parameters are obtained and the multi-factor coupling failure mechanism Φ of the intelligent system is obtained. BRIEF DESCRIPTION OF THE DRAWINGS
[0083] Figure 1 Flowchart of a failure mechanism analysis method for intelligent system reasoning evolution based on defect sample generation
[0084] Figure 2 Schematic diagram of the inference evolution process model
[0085] Figure 3 Schematic diagram of diverse defect data sample construction
[0086] Figure 4 Schematic diagram of adversarial defect data sample construction
[0087] Figure 5 Visual analysis based on Grad-CAM
[0088] Figure 6 Adversarial failure mechanism model diagram Specific implementation methods
[0090] Step 1: Analyze the evolution process of intelligent system reasoning, find out its key influencing factors, and define its failure criteria;
[0091] Step 1: Identify the key factors affecting the reliability of the intelligent system:
[0092] In the reasoning evolution stage, the key factors affecting the reliability of the intelligent system are the diversity and adversarial nature of the samples, denoted as Div and Adv respectively. By generating diverse and adversarial defect data samples and inputting them into the intelligent system for testing, the failure data of the intelligent system can be obtained, such as Figure 2 As shown;
[0093] Step 2: Define the failure criteria of the intelligent system:
[0094] The recognition accuracy of the intelligent system is used to define its failure criterion. The recognition accuracy of the intelligent system in the original test set is used as the benchmark, denoted as ACC s , when the recognition accuracy of the intelligent system in other defect sample data is lower than this standard, it is considered that the intelligent system has failed and the failure data needs to be recorded. The specific calculation formula for the recognition accuracy of the intelligent system is:
[0095]
[0096] Among them, ACC represents the recognition accuracy of the intelligent system in the test data set; TP represents the number of positive samples predicted as positive by the intelligent system; FP represents the number of negative samples predicted as positive by the intelligent system, FN represents the number of positive samples predicted as negative by the intelligent system, and TN represents the number of negative samples predicted as negative by the intelligent system. Based on this, the failure criterion of the intelligent system is defined as:
[0097]
[0098] Among them, ACC T Represents the recognition accuracy of the intelligent system in the Tth defect data sample.
[0099] [Example] Take the intelligent target recognition system based on ResNet-34 as the test object and perform inference evolution modeling. The test set contains 1,000 test samples, which are injected into the intelligent system for testing. The obtained TP=952, TN=21, FP=7, FN=20, so the recognition accuracy of the intelligent system is:
[0100]
[0101] When the recognition accuracy of the intelligent system in other defect data samples is lower than 97.3%, it is considered that the intelligent system has failed and the failure data needs to be recorded.
[0102] Step 2: Generate defect data samples based on the obtained key influencing factors;
[0103] Step 1: Generation of diverse defect samples based on diffusion model:
[0104] 1) Sample diversity metrics
[0105] Combined with the Cosin similarity index to measure sample diversity, the generated samples are represented as a vector Sam, and the diversity of the generated samples is represented by calculating the cosine value between the vectors. The calculation formula is:
[0106]
[0107] Div=1-|Cosin|
[0108] Among them, Cosin is the sample similarity index, Sam1 represents the vector of the original sample, Sam2 represents the vector of the generated defect sample, and Div represents the calculated diversity.
[0109] 2) Generation of diverse defect samples based on diffusion model
[0110] Diversity defect samples are generated using the diffusion model. In the forward phase of the diffusion model, noise is gradually added to the N image samples in the original image data set until the original image samples are completely destroyed into Gaussian noise. The change process is expressed as:
[0111]
[0112] Among them, x1: T represents the state sequence from time step t = 1 to time T, which is a random variable sequence that represents the state evolution of the original sample within a certain time range; x0 represents the state of the original sample at t = 0; x t represents the state of the sample at time t; x t-1 represents the state of the sample at time t-1; and q(x t |x t-1 )for:
[0113]
[0114] Among them, β t is the variance hyperparameter of the Gaussian distribution, satisfying β t ∈(0,1), represents Gaussian distribution, where a = x t is the value of Gaussian distribution, represents the mean, c = β t I represents the current time step x t The covariance matrix of the conditional Gaussian distribution of .
[0115] After obtaining the completely destroyed samples, entropy reduction is performed to gradually convert the noise data into new image data samples. The change process is expressed as:
[0116]
[0117] Among them, p(x 0:T ) represents the joint probability density distribution of the state sequence from t = 0 to T, p(x T ) represents the state probability distribution at time T, that is, the image state after being completely destroyed by Gaussian noise; and p(x t-1 |x t )for:
[0118]
[0119] Among them, μ(x t ,t) represents the mean, which is about x t and time t, ∑(x t ,t) represents the covariance matrix. The same number of defect data samples as the original data set are generated by the diffusion model, and the diversity of the samples generated by the diffusion model is calculated by combining the sample diversity metric Div. The diffusion model is used to generate defect data samples with different Div values, which are input into the intelligent system to test the recognition accuracy of the intelligent system in samples with different diversity values. Combined with the failure criteria defined in step 1, sub-step 2, the failure data is recorded.
[0120] Step 2: Generation of adversarial defect samples based on Fast Gradient Sign Method (FGSM):
[0121] 1) Sample adversarial metrics
[0122] The L2 norm is used to calculate the adversarial nature of the sample. The generated adversarial sample and the original sample are represented as a vector. The adversarial nature calculation formula is expressed as:
[0123]
[0124] Among them, x represents the pixel vector of the original image, y represents the pixel vector of the generated adversarial image, and x i and i are the pixel vectors of the two images at the i-th position, n represents the total number of pixels in the image, and the calculated Adv is the L2 norm.
[0125] 2) Generation of adversarial defect samples based on FGSM
[0126] The calculation formula of the adversarial sample generation process based on FGSM is:
[0127]
[0128] Among them, ε is the perturbation amplitude or perturbation size, sign(·) is the sign function, using the sign of the gradient, J represents the loss function, θ represents the model parameters, and lab is the label corresponding to the image x. represents the gradient of the loss function J with respect to the input x, and y represents the generated adversarial defect sample. While using FGSM, image rotation and flipping transformations are also added to enhance the adversarial nature of the sample:
[0129]
[0130] Among them, y is the adversarial sample after adding disturbance, p is the flip probability of the adversarial sample, q is the rotation probability of the adversarial sample, TF(y) is the flipped sample, and TL(y) is the rotated sample. Use the adversarial metric to calculate the adversarial value Adv of the generated adversarial sample. By generating adversarial defect data samples with different Adv and inputting them into the intelligent system, the recognition accuracy of the intelligent system in different adversarial defect data samples is tested. Combined with the failure criteria defined in step 1, sub-step 2, the failure data is recorded.
[0131] Step 3: Generation of defect samples with diversity and adversarial coupling:
[0132] 1) Generation of diverse defect samples
[0133] Generate diverse defect samples based on the diffusion model of step 2, sub-step 1, and calculate the diversity Div of the samples through the sample diversity metric;
[0134] 2) Increase the adversarial nature of samples
[0135] Based on the diversity samples, combined with the adversarial defect sample generation method proposed in step 2, we generate diverse and adversarially coupled test samples, and calculate the adversarial Adv of the samples in combination with the sample adversarial metrics.
[0136] The generated diverse and adversarially coupled defect samples are input into the intelligent system to test the recognition accuracy of the intelligent system in the coupled defect data samples. Combined with the failure criteria defined in sub-step 2 of step 1, the failure data is recorded.
[0137] [Example] Diversity defect samples generated using the diffusion model are as follows Figure 3 As shown in Figure 2, adversarial samples under different adversarial indicators are as follows: Figure 4 shown.
[0138] Step 3: Visual failure mechanism analysis based on Grad-CAM;
[0139] Through Grad-CAM, back propagation is performed from the output of the intelligent system to obtain the gradient information of the feature layer before the fully connected layer in the intelligent system, and the average gradient of each feature map at this level is obtained. The expression is:
[0140]
[0141] In the formula, is the average gradient of the k-th channel feature map in the c-th output back propagation; Z is the total number of pixels in the feature image; y c is the maximum class output value output by the intelligent system; A represents the activation matrix, i and j are the row index and column index in the activation matrix respectively, k represents the channel index, is the feature map corresponding to the kth channel. The average gradient of each channel feature map is used as the weight to multiply each channel feature map and superimpose them, and then the negative values less than 0 are removed to eliminate the influence of other categories, and the key area used by the network for classification is obtained. Its expression is:
[0142]
[0143] In the formula, represents the class activation matrix obtained by Grad-CAM; ReLU is a nonlinear activation function used to remove negative values. Multiplying with the original input image and performing weighted summation, we can get the target recognition result visualized by the intelligent system, which is expressed as:
[0144]
[0145] Among them, X is the input image data passed through the preprocessing layer of the neural network. From the images generated by Grad-CAM, we observe the changes in the focus area as the diversity and adversarial nature of the test samples change, and explain the reasons for the intelligent system's misrecognition.
[0146] [Example] The visualization results based on Grad-CAM are as follows Figure 5 As shown, it can be seen that as the disturbance increases, the focus area of the intelligent system gradually shifts from the object itself to the edge area of the image, resulting in recognition errors.
[0147] Step 4: Construct the failure mechanism model of the intelligent system based on the agent model;
[0148] Step 1: Construction of single factor failure mechanism model of intelligent system based on agent model:
[0149] Arrange the failure data recorded in step 2, sub-step 1 and sub-step 2, in order of diversity and multi-resistance values from small to large, and record them as:
[0150] ((Div1,ACC1),(Div2,ACC2),……,(Div m ,ACC num_1 ))
[0151] ((Aav1,ACC1),(Adv2,ACC2),……,(Adv m ,ACC num_2 ))
[0152] Among them, (Div i ,ACC i )(i∈(1,2,...,num_1)) represents the sample diversity value and the recognition accuracy of the intelligent system under the diversity sample, num_1 represents the number of defect data sets with different diversities, (Adv i ,ACC i )(i∈(1,2,...,num_2)) represents the sample adversarial index and the corresponding recognition accuracy of the intelligent system under the adversarial sample, and num_2 represents the number of defect data sets with different adversarial characteristics. Using the failure data, a proxy model based on multi-top fitting is constructed to obtain a single factor mechanism model of the intelligent system, which is expressed as
[0153]
[0154] Among them, in is the sample diversity Div or sample adversarial Adv; represents a single factor mechanism model, which is a function of in; n represents the number of failure data; a i (i∈(1,2,...,n)) represents the parameters in the polynomial. The least squares method is used to solve the model, which is:
[0155]
[0156] Among them, δ(a0,…,a n ) represents the sum of the differences between the model and the failure data, f i represents the actual value of the i-th failure data point. Solve for a0, a1, …, a n The value of makes the value of δ minimum. Taking the derivative of both sides of the function, we get:
[0157]
[0158] ......
[0160]
[0161] This gives the normal equations:
[0162]
[0163] Solve the system of equations and calculate a0,a1,…,a n The value of , the single factor failure mechanism model of the intelligent system is obtained like Figure 6 shown.
[0164] Step 2: Construction of multi-factor failure mechanism model of intelligent system based on neural network
[0165] Arrange the failure data recorded in step 3 of step 2 in ascending order of diversity value, and record it as:
[0166] ((Div1,Adv1,ACC1),(Div2,Adv2,ACC2),……,(Div z ,Adv z ,ACC z ))
[0167] Where z is the number of failure data. Based on the failure data obtained from the test and combined with the deep neural network, a failure mechanism model of the intelligent system under multi-factor coupling is constructed, which is expressed as:
[0168] Φ=g((Div i ,Adv i ,ACC i );θ)+ε i
[0169] Among them, g(·) represents the deep neural network model, (Div i ,Adv i ,ACC i ) is the failure data of the intelligent system, θ represents the neural network parameters, ε i is the error term, and Φ represents the multi-factor mechanism model. Using the failure data of the intelligent system, the network parameters are optimized by minimizing the loss function, which is expressed as:
[0170]
[0171] Among them, L min represents the loss function, ACC j Indicates that the diversity and adversarial values are Div j ,Adv j The intelligent system recognition accuracy corresponding to the j-th failure data, Φ j Indicates that the sample diversity and diversity values are Divj ,Adv j The recognition accuracy of the intelligent system predicted by the multi-factor mechanism model is obtained. After training, the optimal network parameters are obtained and the multi-factor coupling failure mechanism Φ of the intelligent system is obtained.
[0172] [Example] Taking the intelligent target recognition system based on ResNet-34 as the test object, a large number of adversarial test samples are generated to obtain the failure data of the intelligent system, and the failure mechanism of the intelligent system under adversarial attacks is obtained. Preliminary tests show that when the disturbance increases to 0.2, the recognition accuracy of the intelligent system is almost 0. Therefore, we set the upper limit of the disturbance to 0.2, and add disturbances to the adversarial samples at intervals of 0.002 to generate 100 groups of different adversarial test samples. The generated different adversarial test samples are input into the intelligent system for testing, and the recognition accuracy of the intelligent system in each adversarial defect data set is obtained, and the failure data is obtained by combining the failure criteria. Its recognition accuracy curve is shown as follows: Figure 6 As shown. By combining the surrogate model based on polynomial fitting with the failure data obtained from the test, the failure mechanism model is obtained as follows:
[0173]
[0174] Where x is the size of the adversarial perturbation.
Claims
1. An intelligent system reasoning evolution failure mechanism analysis method based on defect sample generation, characterized by It includes the following steps: Step 1: Analyze the evolution process of intelligent system reasoning, find out its key influencing factors, and define its failure criteria; Step 1: Identify the key factors affecting the reliability of the intelligent system: In the reasoning evolution stage, the key factors affecting the reliability of the intelligent system are the diversity and adversarial nature of the samples, which are denoted as Div and Adv respectively. By generating diverse and adversarial defect data samples and inputting them into the intelligent system for testing, the failure data of the intelligent system can be obtained. Step 2: Define the failure criteria of the intelligent system: Use the intelligent system recognition accuracy to define its failure criteria; The recognition accuracy of the intelligent system in the original test set is used as the benchmark, denoted as ACC s , when the recognition accuracy of the intelligent system in other defect sample data is lower than the benchmark, it is considered that the intelligent system has failed and the failure data needs to be recorded; the specific calculation formula for the recognition accuracy of the intelligent system is: Among them, ACC represents the recognition accuracy of the intelligent system in the test data set; TP represents the number of positive samples predicted as positive by the intelligent system; FP represents the number of negative samples predicted as positive by the intelligent system, FN represents the number of positive samples predicted as negative by the intelligent system, and TN represents the number of negative samples predicted as negative by the intelligent system. Based on this, the failure criterion of the intelligent system is defined as: Among them, ACC T represents the recognition accuracy of the intelligent system in the Tth defect data sample; Step 2: Generate defect data samples based on the obtained key influencing factors; Step 1: Generate diverse defect samples based on the diffusion model: 1) Sample diversity metrics Combined with the Cosin similarity index to measure sample diversity, the generated samples are represented as a vector Sam, and the diversity of the generated samples is represented by calculating the cosine value between the vectors. The calculation formula is: Div=1-|Cosin| Among them, Cosin is the sample similarity index, Sam1 represents the vector of the original sample, Sam2 represents the vector of the generated defect sample, and Div represents the calculated diversity; 2) Generation of diverse defect samples based on diffusion model Diversity defect samples are generated using the diffusion model. In the forward phase of the diffusion model, noise is gradually added to the N image samples in the original image data set until the original image samples are completely destroyed into Gaussian noise. The change process is expressed as: Among them, x 1:T represents the state sequence from time step t = 1 to time T, which is a random variable sequence that represents the state evolution of the original sample within a certain time range; x0 represents the state of the original sample at t = 0; x t represents the state of the sample at time t; x t-1 represents the state of the sample at time t-1; and q(x t |x t-1 )for: Among them, β t is the variance hyperparameter of the Gaussian distribution, satisfying β t ∈(0,1), represents Gaussian distribution, where a = x t is the value of Gaussian distribution, represents the mean, c = β t I represents the current time step x t The covariance matrix of the conditional Gaussian distribution of ; After obtaining the completely destroyed samples, entropy reduction is performed to gradually convert the noise data into new image data samples. The change process is expressed as: Among them, p(x 0:T ) represents the joint probability density distribution of the state sequence from t = 0 to T, p(x T ) represents the state probability distribution at time T, that is, the image state after being completely destroyed by Gaussian noise; and p(x t-1 |x t )for: Among them, μ(x t ,t) represents the mean, which is about x t and time t,∑(x t , t) represents the covariance matrix; the same number of defect data samples as the original data set are generated by the diffusion model, and the diversity of the samples generated by the diffusion model is calculated by combining the sample diversity measurement index Div; the defect data samples with different Div values are generated by using the diffusion model, and input into the intelligent system to test the recognition accuracy of the intelligent system in samples with different diversity values, and the failure criterion defined in step 1, sub-step 2 is combined to record the failure data; Step 2: Generation of adversarial defect samples based on the Fast Gradient Sign Method (FGSM): 1) Sample adversarial metrics The L2 norm is used to calculate the adversarial nature of the sample. The generated adversarial sample and the original sample are represented as a vector. The adversarial nature calculation formula is expressed as: Among them, x represents the pixel vector of the original image, y represents the pixel vector of the generated adversarial image, and x i and i are the pixel vectors of the two images at the i-th position, n represents the total number of pixels in the image, and the calculated Adv is the L2 norm; 2) Generation of adversarial defect samples based on FGSM The calculation formula of the adversarial sample generation process based on FGSM is: Among them, ε is the perturbation amplitude or perturbation size, sign(·) is the sign function, using the sign of the gradient, J represents the loss function, θ represents the model parameters, and lab is the label corresponding to the image x. represents the gradient of the loss function J with respect to the input x, and y represents the generated adversarial defect sample. While using FGSM, image rotation and flipping transformations are also added to enhance the adversarial nature of the sample: Wherein, y is the adversarial sample after adding disturbance, p represents the flip probability of the adversarial sample, q represents the rotation probability of the adversarial sample, TF(y) represents the flipped sample, and TL(y) represents the rotated sample; the adversarial value Adv of the adversarial sample is calculated using the adversarial metric, and adversarial defect data samples with different Adv are generated and input into the intelligent system to test the recognition accuracy of the intelligent system in different adversarial defect data samples, and the failure data is recorded in combination with the failure criterion defined in step 1, sub-step 2; Step 3: Generation of defect samples with diversity and adversarial coupling: 1) Generation of diverse defect samples Generate diverse defect samples based on the diffusion model of step 2, sub-step 1, and calculate the diversity Div of the samples through the sample diversity metric; 2) Increase the adversarial nature of samples Based on the diversity samples, combined with the adversarial defect sample generation method proposed in step 2, generate diverse and adversarially coupled test samples, and calculate the adversarial Adv of the samples in combination with the sample adversarial metrics; Input the generated diverse and adversarially coupled defect samples into the intelligent system, test the recognition accuracy of the intelligent system in the coupled defect data samples, and record the failure data in combination with the failure criteria defined in step 1, sub-step 2; Step 3: Visual failure mechanism analysis based on Grad-CAM; Through Grad-CAM, back propagation is performed from the output of the intelligent system to obtain the gradient information of the feature layer before the fully connected layer in the intelligent system, and the average gradient of each feature map at this level is obtained. The expression is: In the formula, is the average gradient of the k-th channel feature map in the c-th output back propagation; Z is the total number of pixels in the feature image; y c is the maximum class output value output by the intelligent system; A represents the activation matrix, i and j are the row index and column index in the activation matrix respectively, k represents the channel index, is the feature map corresponding to the kth channel; the average gradient of each channel feature map is used as the weight to multiply each channel feature map and superimpose, and then the negative values less than 0 are removed to eliminate the influence of other categories, and the key area used by the network for classification is obtained, and its expression is: In the formula, Represents the class activation matrix obtained by Grad-CAM; ReLU is a nonlinear activation function used to remove negative values; Multiplying with the original input image and performing weighted summation, we can get the target recognition result visualized by the intelligent system, which is expressed as: Where X is the input image data passed through the preprocessing layer of the neural network; observe the changes in the focus area of the image generated by Grad-CAM as the diversity and adversarial nature of the test samples change, and explain the reasons for the intelligent system's misrecognition; Step 4: Construct the failure mechanism model of the intelligent system based on the agent model; Step 1: Construction of single factor failure mechanism model of intelligent system based on agent model: Arrange the failure data recorded in step 2, sub-step 1 and sub-step 2, in order of diversity and multi-resistance values from small to large, and record them as: ((Div1,ACC1),(Div2,ACC2),……,(Div m ,ACC num_1 )) ((Adv1,ACC1),(Adv2,ACC2),……,(Adv m ,ACC num_2 )) Among them, (Div i ,ACC i )(i∈(1,2,…,num_1)) represents the sample diversity value and the recognition accuracy of the intelligent system under the diversity sample. num_1 represents the number of defect data sets with different diversities. (Adv i ,ACC i )(i∈(1,2,…,num_2)) represents the sample adversarial index and the corresponding recognition accuracy of the intelligent system under the adversarial sample, and mum_2 represents the number of defect data sets with different adversarial characteristics. Using the failure data, a proxy model based on polynomial fitting is constructed to obtain a single factor mechanism model of the intelligent system, which is expressed as Among them, in is the sample diversity Doiv or sample adversarial Adv; represents a single factor mechanism model, which is a function of in; n represents the number of failure data; a i (i∈(1,2,…,n)) represents the parameters in the polynomial. The least squares method is used to solve the model, which is: Among them, δ(a0,…,a n ) represents the sum of the differences between the model and the failure data, f i represents the actual value of the i-th failure data point; solve for a0, a1, …, a n The value of δ is minimized; taking the derivative of both sides of the function, we get: …… This gives the normal equations: Solve the system of equations and calculate a0,a1,…,a n The value of , the single factor failure mechanism model of the intelligent system is obtained Step 2: Construction of multi-factor failure mechanism model of intelligent system based on neural network Arrange the failure data recorded in step 3 of step 2 in ascending order of diversity value, and record it as: ((Div1,Adv1,ACC1),(Div2,Adv2,ACC2),……,(Div z ,Adv z ,ACC z )) Where z is the number of failure data. Based on the failure data obtained from the test and combined with the deep neural network, a failure mechanism model of the intelligent system under multi-factor coupling is constructed, which is expressed as: Φ1g((Div i ,Adv i ,ACC i )zθ)+ε i Among them, g(·) represents the deep neural network model, (Div i ,Adv i ,ACC i ) is the failure data of the intelligent system, θ represents the neural network parameters, ε i is the error term, Φ represents the multi-factor mechanism model; using the failure data of the intelligent system, the network parameters are optimized by minimizing the loss function, which is expressed by the formula: Among them, L min represents the loss function, ACC j Indicates that the diversity and adversarial values are Div j ,Adv j The intelligent system recognition accuracy corresponding to the j-th failure data, Φ j Indicates that the sample diversity and diversity values are Div j ,Adv j The recognition accuracy of the intelligent system predicted by the multi-factor mechanism model is calculated; after training, the optimal network parameters are obtained, and the multi-factor coupling failure mechanism Φ of the intelligent system is obtained.
Citation Information
Patent Citations
Single-step adversarial training method and system based on data enhancement and step length adjustment
CN117171762A
Adversarial sample generation method for black-box face recognition system in physical domain
CN117636440A
Robustness improvement method for track defect detection system
CN117934819A
Image data analysis processing system and method based on artificial intelligence
CN118015246A
Method and system for defending against adversarial sample in image classification, and data processing terminal
US20230022943A1
Cited By
Intelligent identification system failure mechanism analysis method based on Do operator intervention
CN119918647A
A Failure Mechanism Analysis Method for Intelligent Recognition Systems Based on Do Operator Intervention
CN119918647B
