An intelligent system reasoning evolution failure mechanism analysis method based on defect sample generation

Abstract

The present application is directed to the problem that the failure mechanism of intelligent system in reasoning evolution stage is difficult to characterize, and proposes an intelligent system reasoning evolution failure mechanism analysis method based on defect sample generation. It contains four steps: (1) modeling the reasoning evolution process of intelligent system, finding out its key influencing factors, and defining its failure criterion; (2) combining the obtained key influencing factors to generate defect data samples; (3) visualizing the failure mechanism based on Grad-CAM; (4) constructing the failure mechanism model of intelligent system based on the proxy model.

Description

Technical Field

[0001] This invention addresses intelligent systems and proposes a failure mechanism analysis method based on defect sample generation technology. This method models the reasoning evolution process of intelligent systems for typical tasks such as intelligent perception, cognition, and decision-making, determining the failure criteria and key influencing factors of the intelligent model. It then generates data samples using a suitable defect sample generation method, inputting them into the intelligent system for testing to obtain failure data. By correlating the failure data with key influencing factor metrics, a correlation model between the intelligent system's output accuracy and its key influencing factors is obtained based on a surrogate model, thereby revealing the failure mechanism of the intelligent system. This invention belongs to the field of artificial intelligence and reliability engineering. Background Technology

[0002] In recent years, with the continuous development of artificial intelligence technology, an increasing number of intelligent systems based on neural networks as their core algorithms have been widely applied in fields such as autonomous driving, security monitoring, agriculture, and facial recognition. However, due to the unique problems of unclear failure sources and complex failure chains in intelligent systems during use, it is necessary to propose an adapted failure mechanism analysis method that combines the technical characteristics of intelligent systems to provide a basis for the reliability design of intelligent sensing systems.

[0003] Intelligent systems, when performing reasoning tasks such as target detection and recognition, are often affected by various factors and adversarial attacks. To explore the impact of interference, most existing methods analyze the output results of software systems through extensive testing, without providing methods for system failure mechanism analysis and model building. Meanwhile, existing general software system failure mechanism analysis methods only qualitatively describe software failure modes and causes, failing to deeply characterize the model's failure process. The current lack of failure mechanism analysis methods for intelligent systems restricts their high-quality development.

[0004] Based on this, this invention addresses the difficulty in characterizing the failure mechanism of intelligent systems during the reasoning and evolution stage. It proposes a failure mechanism analysis method for intelligent systems based on defect sample generation, which deeply analyzes the causes of failure of intelligent models and provides a theoretical basis for the design improvement of intelligent systems. Summary of the Invention

[0005] This invention addresses the difficulty in characterizing failure mechanisms during the reasoning evolution stage of intelligent systems. It proposes a method for analyzing failure mechanisms in intelligent system reasoning evolution based on defect sample generation. The overall research approach of this invention is as follows: Figure 1 As shown, the specific steps are as follows:

[0006] Step 1: Analyze the reasoning evolution process of the intelligent system, identify its key influencing factors, and define its failure criteria.

[0007] Step 1.1 Identify the key factors affecting the reliability of intelligent systems:

[0008] In the reasoning evolution stage, the key factors affecting the reliability of intelligent systems are sample diversity and adversarial nature, denoted as , respectively. , By generating diverse and adversarial defect data samples, which are then input into the intelligent system for testing, failure data of the intelligent system can be obtained, such as... Figure 2 As shown;

[0009] Step 1.2 Define the failure criteria for the intelligent system:

[0010] The failure criterion is defined using the recognition accuracy of the intelligent system. The recognition accuracy of the intelligent system on the original test set is used as the benchmark, denoted as... If the intelligent system's recognition accuracy in other defective sample data falls below this standard, the intelligent system is considered to have failed, and the failure data must be recorded. The specific formula for calculating the intelligent system's recognition accuracy is as follows:

[0011]

[0012] in, This represents the recognition accuracy of the intelligent system in the test dataset; This represents the number of positive samples predicted as positive by the intelligent system. This represents the number of negative samples that were predicted as positive by the intelligent system. This represents the number of positive samples that were predicted as negative by the intelligent system. This represents the number of negative samples predicted as negative by the intelligent system. Based on this, the failure criterion of the intelligent system is defined as:

[0013]

[0014] in, This represents the accuracy of the intelligent system in identifying the defective data sample T.

[0015] Step 2: Combine the obtained key influencing factors to generate defect data samples.

[0016] Step 2.1 Generation of diverse defect samples based on diffusion model:

[0017] 1) Sample diversity measurement indicators

[0018] Combining the Cosin similarity metric to measure sample diversity, the generated samples are represented as a vector. The diversity of generated samples is represented by calculating the cosine value between vectors, and the calculation formula is as follows:

[0019]

[0020]

[0021] Cosin is a sample similarity index. The vector representing the original sample. A vector representing the generated defective samples. This represents the diversity obtained from the calculation.

[0022] 2) Generation of diverse defect samples based on diffusion model

[0023] A diffusion model is used to generate diverse defect samples. In the forward phase of the diffusion model, noise is gradually added to N image samples in the original image dataset until the original image samples are completely destroyed into Gaussian noise. The process is represented as follows:

[0024]

[0025] in, Indicates time steps The state sequence up to time T is a sequence of random variables, representing the state evolution of the original sample within a certain time range; express The state of the original sample at that time; This represents the state of the sample at time t; express The state of the sample at any given time; and for:

[0026]

[0027] in, Let be the variance hyperparameter of the Gaussian distribution, satisfying , Denotes a Gaussian distribution, where Let be the values ​​of the Gaussian distribution. This represents the mean. Indicates the current time step The covariance matrix of a conditional Gaussian distribution.

[0028] After obtaining the completely destroyed sample, entropy reduction is then performed to gradually convert the noisy data into new image data samples. This process can be represented as follows:

[0029]

[0030] in, express The joint probability density distribution of the state sequence up to time T. This represents the state probability distribution at time T, i.e., the image state after it has been completely destroyed by Gaussian noise; while for:

[0031] )

[0032] in, The mean is about and time The function, This represents the covariance matrix. A diffusion model is used to generate the same number of defect data samples as the original dataset, combined with a sample diversity metric. The diversity of samples generated by the diffusion model was calculated. The diffusion model was used to generate samples with different... The defective data samples are input into the intelligent system to test the recognition accuracy of the intelligent system in samples with different values. Combined with the failure criteria defined in sub-step 1.2 of step one, the failure data is recorded.

[0033] Step 2.2 Generation of adversarial defect samples based on Fast Gradient Sign Method (FGSM):

[0034] 1) Sample adversarial metrics

[0035] Using the L2 norm to calculate the adversarial nature of a sample, the generated adversarial sample and the original sample are represented as a vector. The formula for calculating the adversarial nature is as follows:

[0036]

[0037] in, Represents the pixel vector of the original image. This represents the pixel vector of the generated adversarial image. and Let be the pixel vectors at position i in the two images, and let n represent the total number of pixels in the image. The calculated... This is the L2 norm.

[0038] 2) Generation of adversarial defect samples based on FGSM

[0039] The calculation formula for the adversarial example generation process based on FGSM is as follows:

[0040]

[0041] in, The magnitude or size of the disturbance. It is a sign function, using the sign of the gradient. Represents the loss function. This represents the model parameters. It is an image The corresponding tags Represents the loss function For input gradient, This represents the generated adversarial flawed samples. Image rotation and flipping transformations were added along with FGSM to enhance the adversarial nature of the samples:

[0042]

[0043] in, For adversarial examples with added perturbations. Indicates the probability of adversarial sample flipping. This represents the rotation probability of the adversarial example. The sample after being flipped. This represents the rotated sample. The adversarial value of the generated adversarial example is calculated using an adversarial metric. By generating different The adversarial defect data samples are input into the intelligent system to test the recognition accuracy of the intelligent system in different adversarial defect data samples. Combined with the failure criteria defined in sub-step 1.2 of step one, the failure data is recorded.

[0044] Step 2.3 Generation of defect samples with diversity and adversarial coupling:

[0045] 1) Generation of diverse defect samples

[0046] Based on the diffusion model generated in sub-step 2.1 of step two, the diversity of defect samples is calculated using a sample diversity metric. ;

[0047] 2) Increase the adversarial nature of the samples

[0048] Based on the diverse samples, and combined with the adversarial defect sample generation method proposed in sub-step 2.2 of step two, diverse and adversarially coupled test samples are generated. Then, the adversarial nature of the samples is calculated using the sample adversarial metric. .

[0049] The generated diverse and adversarially coupled defect samples are input into the intelligent system to test the system's recognition accuracy in the coupled defect data samples. The failure data is recorded in conjunction with the failure criteria defined in sub-step 1.2 of step one.

[0050] Step 3: Visualized failure mechanism analysis based on Grad-CAM.

[0051] Grad-CAM is used to backpropagate from the output of the intelligent system to obtain the gradient information of the feature layer preceding the fully connected layer in the intelligent system, and the average gradient of each feature map at that layer is calculated. The expression for this gradient is as follows:

[0052]

[0053] In the formula, Z is the average gradient of the feature map of the k-th channel during the backpropagation of the c-th output; Z is the total number of pixels in the feature image. The maximum class output value of the intelligent system; A represents the activation matrix, i and j are the row and column indices in the activation matrix, respectively, and k represents the channel index. Let be the feature map corresponding to the k-th channel. The average gradient of each channel's feature map is used as a weight to multiply the feature maps of each channel and then summed. Negative values ​​less than 0 are then removed to eliminate the influence of other categories, resulting in the key region used by the network for classification. Its expression is:

[0054]

[0055] In the formula, This represents the class activation matrix obtained by Grad-CAM; This is a non-linear activation function used to remove negative values. By... Multiplying the result by the original input image and performing a weighted sum yields the visualized target recognition result of the intelligent system, expressed as:

[0056]

[0057] in, The input image data is passed through the preprocessing layer of the neural network. The changes in the region of interest are observed in the images generated by Grad-CAM as the diversity and adversarial nature of the test samples change, such as... Figure 5 As shown, this explains the reasons for the intelligent system's incorrect identification.

[0058] Step 4: Construct a failure mechanism model for the intelligent system based on the agent model.

[0059] Step 4.1 Construction of a single-factor failure mechanism model for intelligent systems based on the surrogate model:

[0060] The failure data recorded in sub-steps 2.1 and 2.2 of step two are arranged in ascending order of diversity and multi-resistance values, and denoted as follows:

[0061]

[0062]

[0063] in, This represents the sample diversity values ​​and the corresponding recognition accuracy of the intelligent system under that diverse sample. This represents the number of defect datasets with different levels of diversity. This represents the adversarial index of a sample and the corresponding recognition accuracy of the intelligent system under that adversarial sample. This represents the number of defect datasets with different adversarial strengths. Using the failure data, a surrogate model based on multinomial fitting is constructed, resulting in a single-factor mechanism model of the intelligent system, denoted as...

[0064]

[0065] in, For sample diversity or sample adversarial ; This represents a single-factor mechanism model, which is about The function; n represents the number of invalid data; The parameters in the polynomial are represented by the following expression. The model is solved using the least squares method, and its formula is:

[0066]

[0067] in, This represents the sum of the differences between the model and the failure data. Let represent the actual value of the i-th failed data point. Solve... The value of makes The value of is minimized. Taking the derivative of both sides of the function, we get:

[0068]

[0069]

[0070]

[0071]

[0072] The normal system of equations is derived as follows:

[0073]

[0074] Solve the system of equations and calculate The value is used to obtain the single-factor failure mechanism model of the intelligent system. .

[0075] Step 4.2 Construction of a Multi-Factor Failure Mechanism Model for Intelligent Systems Based on Neural Networks

[0076] Arrange the failure data recorded in sub-step 2.3 of step two in ascending order of diversity values, and denote them as follows:

[0077]

[0078] Where z represents the number of failure data points. Based on the failure data obtained from testing, and combined with a deep neural network, a failure mechanism model of the intelligent system under multi-factor coupling is constructed, expressed as:

[0079]

[0080] in, Represents a deep neural network model. For intelligent system failure data, Represents the parameters of the neural network. For error terms, This represents a multi-factor mechanism model. Using failure data from intelligent systems, the network parameters are optimized by minimizing a loss function, expressed by the following formula:

[0081]

[0082] in, Represents the loss function. The values ​​representing diversity and antagonism are respectively The The accuracy rate of intelligent system identification for each failure data point This indicates that the sample diversity and diversity values ​​are respectively The recognition accuracy of intelligent systems predicted by a multi-factor mechanism model is obtained. Optimal network parameters are obtained through training, leading to the understanding of the multi-factor coupling failure mechanism of the intelligent system. . Attached Figure Description

[0083] Figure 1. Flowchart of a method for analyzing the failure mechanism of intelligent system inference evolution based on defect sample generation.

[0084] Figure 2. Schematic diagram of the reasoning evolution process model

[0085] Figure 3. Schematic diagram of the construction of diverse defect data samples

[0086] Figure 4. Schematic diagram of adversarial defect data sample construction

[0087] Figure 5. Visualization analysis based on Grad-CAM

[0088] Figure 6. Diagram of the adversarial failure mechanism Specific implementation methods

[0089] Step 1: Analyze the reasoning evolution process of the intelligent system, identify its key influencing factors, and define its failure criteria;

[0090] Step 1.1: Identify the key factors affecting the reliability of intelligent systems:

[0091] In the reasoning evolution stage, the key factors affecting the reliability of intelligent systems are sample diversity and adversarial nature, denoted as , respectively. , By generating diverse and adversarial defect data samples, which are then input into the intelligent system for testing, failure data of the intelligent system can be obtained, such as... Figure 2 As shown;

[0092] Step 1.2: Define the failure criteria for the intelligent system:

[0093] The failure criterion is defined using the recognition accuracy of the intelligent system. The recognition accuracy of the intelligent system on the original test set is used as the benchmark, denoted as... If the intelligent system's recognition accuracy in other defective sample data falls below this standard, the intelligent system is considered to have failed, and the failure data must be recorded. The specific formula for calculating the intelligent system's recognition accuracy is as follows:

[0094]

[0095] in, This represents the recognition accuracy of the intelligent system in the test dataset; This represents the number of positive samples predicted as positive by the intelligent system. This represents the number of negative samples that were predicted as positive by the intelligent system. This represents the number of positive samples that were predicted as negative by the intelligent system. This represents the number of negative samples predicted as negative by the intelligent system. Based on this, the failure criterion of the intelligent system is defined as:

[0096]

[0097] in, This represents the accuracy of the intelligent system in identifying the defective data sample T.

[0098]

Example

[0099]

[0100] When the accuracy rate of the intelligent system in identifying other defective data samples is lower than 97.3%, the intelligent system is considered to have failed and the failure data needs to be recorded.

[0101] Step 2: Combine the obtained key influencing factors to generate defect data samples;

[0102] Step 2.1: Generation of diverse defect samples based on diffusion model:

[0103] 1) Sample diversity measurement indicators

[0104] Combining the Cosin similarity metric to measure sample diversity, the generated samples are represented as a vector. The diversity of generated samples is represented by calculating the cosine value between vectors, and the calculation formula is as follows:

[0105]

[0106]

[0107] Cosin is a sample similarity index. The vector representing the original sample. A vector representing the generated defective samples. This represents the diversity obtained from the calculation.

[0108] 2) Generation of diverse defect samples based on diffusion model

[0109] A diffusion model is used to generate diverse defect samples. In the forward phase of the diffusion model, noise is gradually added to N image samples in the original image dataset until the original image samples are completely destroyed into Gaussian noise. The process is represented as follows:

[0110]

[0111] in, Indicates time steps The state sequence up to time T is a sequence of random variables, representing the state evolution of the original sample within a certain time range; express The state of the original sample at that time; This represents the state of the sample at time t; express The state of the sample at any given time; and for:

[0112]

[0113] in, Let be the variance hyperparameter of the Gaussian distribution, satisfying , Denotes a Gaussian distribution, where Let be the values ​​of the Gaussian distribution. This represents the mean. Indicates the current time step The covariance matrix of a conditional Gaussian distribution.

[0114] After obtaining the completely destroyed sample, entropy reduction is then performed to gradually convert the noisy data into new image data samples. This process can be represented as follows:

[0115]

[0116] in, express The joint probability density distribution of the state sequence up to time T. This represents the state probability distribution at time T, i.e., the image state after it has been completely destroyed by Gaussian noise; while for:

[0117] )

[0118] in, The mean is about and time The function, This represents the covariance matrix. A diffusion model is used to generate the same number of defect data samples as the original dataset, combined with a sample diversity metric. The diversity of samples generated by the diffusion model was calculated. The diffusion model was used to generate samples with different... The defective data samples are input into the intelligent system to test the recognition accuracy of the intelligent system in samples with different values. Combined with the failure criteria defined in sub-step 1.2 of step one, the failure data is recorded.

[0119] Step 2.2: Generation of adversarial defect samples based on Fast Gradient Sign Method (FGSM):

[0120] 1) Sample adversarial metrics

[0121] Using the L2 norm to calculate the adversarial nature of a sample, the generated adversarial sample and the original sample are represented as a vector. The formula for calculating the adversarial nature is as follows:

[0122]

[0123] in, Represents the pixel vector of the original image. This represents the pixel vector of the generated adversarial image. and Let be the pixel vectors at position i in the two images, and let n represent the total number of pixels in the image. The calculated... This is the L2 norm.

[0124] 2) Generation of adversarial defect samples based on FGSM

[0125] The calculation formula for the adversarial example generation process based on FGSM is as follows:

[0126]

[0127] in, The magnitude or size of the disturbance. It is a sign function, using the sign of the gradient. Represents the loss function. This represents the model parameters. It is an image The corresponding tags Represents the loss function For input gradient, This represents the generated adversarial flawed samples. Image rotation and flipping transformations were added along with FGSM to enhance the adversarial nature of the samples:

[0128]

[0129] in, For adversarial examples with added perturbations. Indicates the probability of adversarial sample flipping. This represents the rotation probability of the adversarial example. The sample after being flipped. This represents the rotated sample. The adversarial value of the generated adversarial example is calculated using an adversarial metric. By generating different The adversarial defect data samples are input into the intelligent system to test the recognition accuracy of the intelligent system in different adversarial defect data samples. Combined with the failure criteria defined in sub-step 1.2 of step one, the failure data is recorded.

[0130] Step 2.3: Generation of defect samples with diversity and adversarial coupling:

[0131] 1) Generation of diverse defect samples

[0132] Based on the diffusion model generated in sub-step 2.1 of step two, the diversity of defect samples is calculated using a sample diversity metric. ;

[0133] 2) Increase the adversarial nature of the samples

[0134] Based on the diverse samples, and combined with the adversarial defect sample generation method proposed in sub-step 2.2 of step two, diverse and adversarially coupled test samples are generated. Then, the adversarial nature of the samples is calculated using the sample adversarial metric. .

[0135] The generated diverse and adversarially coupled defect samples are input into the intelligent system to test the system's recognition accuracy in the coupled defect data samples. The failure data is recorded in conjunction with the failure criteria defined in sub-step 1.2 of step one.

[0136] [Example] Diversity defect samples generated using a diffusion model, such as... Figure 3 As shown, adversarial samples under different adversarial indices are as follows: Figure 4 As shown.

[0137] Step 3: Visualized failure mechanism analysis based on Grad-CAM;

[0138] Grad-CAM is used to backpropagate from the output of the intelligent system to obtain the gradient information of the feature layer preceding the fully connected layer in the intelligent system, and the average gradient of each feature map at that layer is calculated. The expression for this gradient is as follows:

[0139]

[0140] In the formula, Z is the average gradient of the feature map of the k-th channel during the backpropagation of the c-th output; Z is the total number of pixels in the feature image. The maximum class output value of the intelligent system; A represents the activation matrix, i and j are the row and column indices in the activation matrix, respectively, and k represents the channel index. Let be the feature map corresponding to the k-th channel. The average gradient of each channel's feature map is used as a weight to multiply the feature maps of each channel and then summed. Negative values ​​less than 0 are then removed to eliminate the influence of other categories, resulting in the key region used by the network for classification. Its expression is:

[0141]

[0142] In the formula, This represents the class activation matrix obtained by Grad-CAM; This is a non-linear activation function used to remove negative values. By... Multiplying the result by the original input image and performing a weighted sum yields the visualized target recognition result of the intelligent system, expressed as:

[0143]

[0144] in, The input image data is passed through the preprocessing layer of the neural network. The changes in the region of interest in the images generated by Grad-CAM with variations in the diversity and adversarial nature of the test samples are observed to explain the reasons for the intelligent system's misidentification.

[0145] [Example] Visualization results obtained based on Grad-CAM are as follows: Figure 5 As shown, it can be seen that as the disturbance increases, the focus of the intelligent system gradually shifts from the object itself to the edge region of the image, resulting in recognition errors.

[0146] Step 4: Construct a failure mechanism model for the intelligent system based on the agent model;

[0147] Step 4.1: Construction of a single-factor failure mechanism model for intelligent systems based on the surrogate model:

[0148] The failure data recorded in sub-steps 2.1 and 2.2 of step two are arranged in ascending order of diversity and multi-resistance values, and denoted as follows:

[0149]

[0150]

[0151] in, This represents the sample diversity values ​​and the corresponding recognition accuracy of the intelligent system under that diverse sample. This represents the number of defect datasets with different levels of diversity. This represents the adversarial index of a sample and the corresponding recognition accuracy of the intelligent system under that adversarial sample. This represents the number of defect datasets with different adversarial strengths. Using the failure data, a surrogate model based on multinomial fitting is constructed, resulting in a single-factor mechanism model of the intelligent system, denoted as...

[0152]

[0153] in, For sample diversity or sample adversarial ; This represents a single-factor mechanism model, which is about The function; n represents the number of invalid data; The parameters in the polynomial are represented by the following expression. The model is solved using the least squares method, and its formula is:

[0154]

[0155] in, This represents the sum of the differences between the model and the failure data. Let represent the actual value of the i-th failed data point. Solve... The value of makes The value of is minimized. Taking the derivative of both sides of the function, we get:

[0156]

[0157]

[0158]

[0159]

[0160] The normal system of equations is derived as follows:

[0161]

[0162] Solve the system of equations and calculate The value is used to obtain the single-factor failure mechanism model of the intelligent system. ,like Figure 6 As shown.

[0163] Step 4.2: Construction of a Multi-Factor Failure Mechanism Model for Intelligent Systems Based on Neural Networks

[0164] Arrange the failure data recorded in sub-step 2.3 of step two in ascending order of diversity values, and denote them as follows:

[0165]

[0166] Where z represents the number of failure data points. Based on the failure data obtained from testing, and combined with a deep neural network, a failure mechanism model of the intelligent system under multi-factor coupling is constructed, expressed as:

[0167]

[0168] in, Represents a deep neural network model. For intelligent system failure data, Represents the parameters of the neural network. For error terms, This represents a multi-factor mechanism model. Using failure data from intelligent systems, the network parameters are optimized by minimizing a loss function, expressed by the following formula:

[0169]

[0170] in, Represents the loss function. The values ​​representing diversity and antagonism are respectively The The accuracy rate of intelligent system identification for each failure data point This indicates that the sample diversity and diversity values ​​are respectively The recognition accuracy of intelligent systems predicted by a multi-factor mechanism model is obtained. Optimal network parameters are obtained through training, leading to the understanding of the multi-factor coupling failure mechanism of the intelligent system. .

[0171] [Example] Using a ResNet-34-based intelligent target recognition system as the test object, failure data of the intelligent system was obtained by generating a large number of adversarial test samples to understand the failure mechanism of the intelligent system under adversarial attacks. Preliminary testing revealed that when the perturbation increased to 0.2, the recognition accuracy of the intelligent system was almost zero. Therefore, we set the perturbation upper limit to 0.2 and added perturbations to the adversarial samples at intervals of 0.002, generating 100 sets of test samples with different adversarial levels. The generated test samples with different adversarial levels were input into the intelligent system for testing, obtaining the recognition accuracy of the intelligent system in various adversarial vulnerability datasets. Combined with the failure criteria, the failure data was obtained. The recognition accuracy curve is shown below. Figure 6 As shown. The failure mechanism model obtained by combining a surrogate model based on polynomial fitting with failure data obtained from testing is as follows:

[0172]

[0173] in, To counteract the magnitude of the disturbance.

Claims

1. A method for analyzing the failure mechanism of intelligent system reasoning evolution based on defect sample generation, characterized in that... It includes the following steps: Step 1: Analyze the reasoning evolution process of the intelligent system, identify its key influencing factors, and define its failure criteria; Step 1.1 Identify the key factors affecting the reliability of intelligent systems: In the reasoning evolution stage, the key factors affecting the reliability of intelligent systems are sample diversity and adversarial nature, denoted as , respectively. , By generating diverse and adversarial defect data samples, and inputting them into the intelligent system for testing, failure data of the intelligent system can be obtained. Step 1.2 Define the failure criteria for the intelligent system: The failure criterion is defined by the accuracy of the intelligent system's identification. The accuracy rate of the intelligent system in recognizing data on the original test set is used as a benchmark, denoted as... If the intelligent system's recognition accuracy in other defective sample data falls below this benchmark, the intelligent system is considered to have failed, and the failure data must be recorded. The specific formula for calculating the intelligent system's recognition accuracy is as follows: in, This represents the recognition accuracy of the intelligent system in the test dataset; This represents the number of positive samples predicted as positive by the intelligent system. This represents the number of negative samples that were predicted as positive by the intelligent system. This represents the number of positive samples that were predicted as negative by the intelligent system. This represents the number of negative samples predicted as negative by the intelligent system; based on this, the failure criterion of the intelligent system is defined as: in, This represents the accuracy of the intelligent system in identifying the T-th defect data sample; Step 2: Combine the obtained key influencing factors to generate defect data samples; Step 2.1 Generation of diverse defect samples based on diffusion model: 1) Sample diversity measurement indicators Combining the Cosin similarity metric to measure sample diversity, the generated samples are represented as a vector. The diversity of generated samples is represented by calculating the cosine value between vectors, and the calculation formula is as follows: Cosin is a sample similarity index. The vector representing the original sample. A vector representing the generated defective samples. This represents the calculated diversity; 2) Generation of diverse defect samples based on diffusion model A diffusion model is used to generate diverse defect samples. In the forward phase of the diffusion model, noise is gradually added to N image samples in the original image dataset until the original image samples are completely destroyed into Gaussian noise. The process is represented as follows: in, Indicates time steps The state sequence up to time T is a sequence of random variables, representing the state evolution of the original sample within a certain time range; express The state of the original sample at that time; This represents the state of the sample at time t; express The state of the sample at any given time; and for: in, Let be the variance hyperparameter of the Gaussian distribution, satisfying , Denotes a Gaussian distribution, where Let be the values ​​of the Gaussian distribution. This represents the mean. Indicates the current time step The covariance matrix of the conditional Gaussian distribution; After obtaining the completely destroyed sample, entropy reduction is then performed to gradually convert the noisy data into new image data samples. This process can be represented as follows: in, express The joint probability density distribution of the state sequence up to time T. This represents the state probability distribution at time T, i.e., the image state after it has been completely destroyed by Gaussian noise; while for: ) in, The mean is about and time The function, Represent the covariance matrix; generate the same number of defect data samples as the original dataset using a diffusion model, combined with a sample diversity metric. The diversity of samples generated by the diffusion model was calculated; the diffusion model was used to generate samples with different... The defective data samples are input into the intelligent system to test the recognition accuracy of the intelligent system in samples with different diversity values. Combined with the failure criteria defined in sub-step 1.2 of step one, the failure data is recorded. Step 2.2 Generation of adversarial defect samples based on Fast Gradient Sign Method (FGSM): 1) Sample adversarial metrics Using the L2 norm to calculate the adversarial nature of a sample, the generated adversarial sample and the original sample are represented as a vector. The formula for calculating the adversarial nature is as follows: in, Represents the pixel vector of the original image. This represents the pixel vector of the generated adversarial image. and Let be the pixel vectors at position i in the two images, and let n represent the total number of pixels in the image. The calculated... That is, the L2 norm; 2) Generation of adversarial defect samples based on FGSM The calculation formula for the adversarial example generation process based on FGSM is as follows: in, The magnitude or size of the disturbance. It is a sign function, using the sign of the gradient. Represents the loss function. This represents the model parameters. It is an image The corresponding tags Represents the loss function For input gradient, This represents the generated adversarial flawed samples; in addition to using FGSM, image rotation and flip transformations were added to enhance the adversarial nature of the samples: in, For adversarial examples with added perturbations. Indicates the probability of adversarial sample flipping. This represents the rotation probability of the adversarial example. The sample after being flipped. This represents the rotated sample; the adversarial value of the generated adversarial example is calculated using an adversarial metric. By generating different The adversarial defect data samples are input into the intelligent system to test the recognition accuracy of the intelligent system in different adversarial defect data samples. Combined with the failure criteria defined in sub-step 1.2 of step one, the failure data is recorded. Step 2.3 Generation of defect samples with diversity and adversarial coupling: 1) Generation of diverse defect samples Based on the diffusion model generated in sub-step 2.1 of step two, the diversity of defect samples is calculated using a sample diversity metric. ; 2) Increase the adversarial nature of the samples Based on the diverse samples, and combined with the adversarial defect sample generation method proposed in sub-step 2.2 of step two, diverse and adversarially coupled test samples are generated. Then, the adversarial nature of the samples is calculated using the sample adversarial metric. ; The generated diverse and adversarially coupled defect samples are input into the intelligent system to test the recognition accuracy of the intelligent system in the coupled defect data samples. Combined with the failure criteria defined in sub-step 1.2 of step one, the failure data is recorded. Step 3: Visualized failure mechanism analysis based on Grad-CAM; Grad-CAM is used to backpropagate from the output of the intelligent system to obtain the gradient information of the feature layer preceding the fully connected layer in the intelligent system, and the average gradient of each feature map in the feature layer preceding the fully connected layer is calculated. The expression for this average gradient is as follows: In the formula, Z is the average gradient of the feature map of the k-th channel during the backpropagation of the c-th output; Z is the total number of pixels in the feature image. The maximum class output value of the intelligent system; A represents the activation matrix, i and j are the row and column indices in the activation matrix, respectively, and k represents the channel index. Let be the feature map corresponding to the k-th channel; multiply the average gradient of each channel's feature map by the feature map of each channel as a weight and then sum them up. Then, remove negative values ​​less than 0 to eliminate the influence of other categories, obtaining the key region used by the network for classification. Its expression is: In the formula, This represents the class activation matrix obtained by Grad-CAM; This is a nonlinear activation function used to remove negative values; by... Multiplying the result by the original input image and performing a weighted sum yields the visualized target recognition result of the intelligent system, expressed as: in, The input image data is passed through the preprocessing layer of the neural network; the changes in the region of interest in the Grad-CAM generated image are observed as the diversity and adversarial nature of the test samples change, and the reasons for the intelligent system's misidentification are explained. Step 4: Construct a failure mechanism model for the intelligent system based on the agent model; Step 4.1 Construction of a single-factor failure mechanism model for intelligent systems based on the surrogate model: The failure data recorded in sub-steps 2.1 and 2.2 of step two are arranged in ascending order of diversity and multi-resistance values, and denoted as follows: in, This represents the sample diversity values ​​and the corresponding recognition accuracy of the intelligent system under that diverse sample. This represents the number of defect datasets with different levels of diversity. This represents the adversarial index of a sample and the corresponding recognition accuracy of the intelligent system under that adversarial sample. This represents the number of defect datasets with different adversarial strengths; using the failure data, a surrogate model based on multinomial fitting is constructed, resulting in a single-factor mechanism model of the intelligent system, denoted as... in, For sample diversity or sample adversarial ; This represents a single-factor mechanism model, which is about The function; n represents the number of invalid data; The parameters in the polynomial are represented by the following expression. The model is solved using the least squares method, and its formula is: in, This represents the sum of the differences between the model and the failure data. Represent the actual value of the i-th failed data point; solve The value of makes The value of is minimized; differentiating both sides of the function, we get: The normal system of equations is derived as follows: Solve the system of equations and calculate The value is used to obtain the single-factor failure mechanism model of the intelligent system. ; Step 4.2 Construction of a Multi-Factor Failure Mechanism Model for Intelligent Systems Based on Neural Networks Arrange the failure data recorded in sub-step 2.3 of step two in ascending order of diversity values, and denote them as follows: Where z represents the number of failure data points; based on the failure data obtained from testing, and combined with a deep neural network, a failure mechanism model of the intelligent system under multi-factor coupling is constructed, expressed as: in, Represents a deep neural network model. For intelligent system failure data, Represents the parameters of the neural network. For error terms, This represents a multi-factor mechanism model; using failure data from intelligent systems, the network parameters are optimized by minimizing the loss function, which is expressed by the following formula: in, Represents the loss function. The values ​​representing diversity and antagonism are respectively The The accuracy rate of intelligent system identification for each failure data point This indicates that the sample diversity and diversity values ​​are respectively The accuracy of intelligent system recognition predicted by the multi-factor mechanism model; after training, the optimal network parameters are obtained, and the multi-factor coupling failure mechanism of the intelligent system is obtained. .

Images