Intelligent system development modeling failure mechanism analysis method based on Bayesian network
Through Bayesian network analysis of sample defect factors in intelligent software development, a quantitative description model is built, which solves the problem of data defect impact in intelligent software development, improves model quality and construction efficiency, and reduces costs.
Patent Information
- Application Number
- CN202411156292.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-06
AI Technical Summary
It is difficult for the existing technology to effectively analyze and screen data defects in the intelligent software development and modeling stage, resulting in the impact of the quality of intelligent models, and the cost of building large-scale data sets is high, and the efficiency of screening defect factors is low.
Using intelligent system based on Bayesian network to develop modeling failure mechanism analysis method, discrete data, establish Bayesian network model, analyze causal relationships, determine the sensitivity of sample defect factors, build a quantitative description model, and optimize the data set construction process to improve model quality.
It improves the efficiency of data set construction and the effect of intelligent models, identify key sample defect factors, reduces the cost of data set construction, and enhances the reliability and accuracy of the model.
Smart Images

Figure CN119089142A_ABST
Abstract
Description
Technical Field
[0001] This paper proposes a Bayesian network-based method for analyzing failure mechanisms in intelligent system development modeling. This method is suitable for defect analysis during the intelligent software development modeling phase. It assesses the impact of different data defects on intelligent software and establishes a quantitative model describing how defect factors influence intelligent software failures. This invention belongs to the field of artificial intelligence. Background Art
[0002] With the continuous advancement of information technology and computer applications, artificial intelligence (AI) technology is being widely applied in various fields. As the vehicle for AI applications, intelligent software, along with its associated potential for failure, has sparked increasing discussion. Intelligent models, the brains of intelligent software, have also posed a significant challenge, raising concerns about the reliability of their inference results. The reliability of intelligent models must not only ensure that requirements are correctly implemented but also that the models do not produce negative effects or abnormal consequences, thereby ensuring that the system's operational functionality is not affected. Otherwise, intelligent model failures can occur. Intelligent models learn experience and knowledge from a large number of training samples to complete their tasks. The quality of these training samples significantly impacts the quality of the intelligent models.
[0003] Internal model factors that impact system reliability stem from dataset construction and design during the intelligent model development phase, which is closely related to the intelligent model's feature engineering. Most existing research on factors influencing the feature engineering phase focuses on specific research questions and outcomes, exploring appropriate feature engineering methods to obtain higher-quality sample datasets for training better models. This paper analyzes how the specific feature engineering process influences model training outcomes and the extent of its impact, providing guidance for dataset construction and design.
[0004] Currently, the sample defect factors that affect the quality of intelligent models mainly include sample balance, sample noise, sample annotation quality, and other feature engineering methods. When users construct data sets, they currently select data with balanced samples, low noise, and high annotation quality. However, for large-scale data sets, the cost of screening all defect factors is too high. The present invention obtains the sensitivity of different defects through sensitivity analysis. During the data construction process, it focuses on eliminating defect factors with high sensitivity, thereby improving the efficiency of data set construction and the effect of intelligent models.
[0005] On this basis, the present invention proposes a sensitivity analysis method for possible data defects in the intelligent software data modeling stage, and establishes a quantitative description model for the impact of defect factors on intelligent model failures. Summary of the Invention
[0006] This paper proposes a Bayesian network-based method for analyzing failure mechanisms in intelligent system development modeling. This method establishes a quantitative description model of how sample defect factors affect model failures and identifies the key factors that influence sample data defects in intelligent models. Specifically, it includes the following steps:
[0007] Step 1: Collect and obtain the original dataset Y related to the task, determine the N sample defect factors of interest, and discretize the N defect factors respectively, including the following steps:
[0008] Step 1.1: Select one of the discretization methods, including equidistance, binarization, and K-means, to discretize the original data;
[0009] Step 1.2: Use the discretized data to build a Bayesian network model;
[0010] Step 1.3: Use the Bayesian network model to reason and analyze the causal relationship between variables within the system;
[0011] Step 1.4: Determine whether the pre-set inference accuracy is met. If so, return to step 1.1 and increase the discretization state value by 1. If not, output the maximum number of discretization states.
[0012] Step 2: Determine the evaluation indicators of the model, establish a Bayesian network between sample defects and evaluation indicators, and express sample defects as a Bayesian network structure and parameter constraints, including normative constraints, interval constraints, intra-distribution constraints, and inter-distribution constraints, as follows:
[0013] The normative constraint means that the sum of all parameters when the parent node probability has the same value is 1, as shown in formula (1):
[0014]
[0015] θ jk Indicates the probability value of the kth child node of the jth parent node in the network.
[0016] Interval constraints indicate that the parameters belong to a given interval, as shown in formula (2):
[0017] 0≤α jk ≤θ jk ≤β jk ≤1 (2)
[0018] α jk and β jk Represents the child node θ jk The probability of is bounded.
[0019] The intra-distribution constraint represents the size relationship between parameters within the same distribution when the parent nodes are the same, as shown in formula (3):
[0020]
[0021] θ jk and θ jk' They represent the probability values of two different child nodes under the same parent node.
[0022] The inter-distribution constraint represents the relationship between the parameters of different distributions when the probability values of the child nodes are the same, as shown in formula (4):
[0023]
[0024] ν j'k and ν jk Indicates the probability of two different parent nodes having the same child node value.
[0025] Step 3: Perform sensitivity analysis on defects based on Bayesian network. The specific steps are as follows:
[0026] Step 3.1: Define the response function of each defect and evaluation index F(x) = g(X), X = x1, x2, ..., x n , define the response function to the defect factor x i The partial derivative of the response function to the defect factor x i Sensitivity S i , as shown in formula (5).
[0027]
[0028] When S i > 0, F(x) changes with x i increases with the increase of S i <0, F(x) increases with x i decreases with the increase of S i = 0, F(x) does not change with x i S i The absolute value reflects the change of F(x) with x. i The speed of change, that is, the function F(x) to the defect factor x i The sensitivity of each defect factor x can be obtained based on the conditional probability change degree of the Bayesian network. i The contribution of the value change to the change of the response function F(x).
[0029] Step 3.2: Determine the variation range of each defect factor, and select the i-th defect factor to construct the j-th sample data set Y according to the determined number of discretized states. ij .
[0030] Step 3.3: Input the constructed dataset into the intelligent system until the intelligent model converges, and use the same test set to test and obtain the dataset Y ij Evaluation index F ij .
[0031] Repeat steps 3.2-3.3 until all defect sample data sets have been tested.
[0032] Step 3.4: Input all the results into the Bayesian network, calculate the sensitivity index of each defect factor, and determine the contribution of each defect factor to the change of the model evaluation index. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 This is an overall flow chart of the failure mechanism analysis method for intelligent system development modeling based on Bayesian networks.
[0034] Figure 2 This is a flowchart of the discretization of defect factors
[0035] Figure 3 This is an example of a cat and dog recognition training set
[0036] Figure 4 It is the Bayesian network of defect factors and evaluation indicators DETAILED DESCRIPTION
[0037] This invention aims at analyzing the failure mechanism in the data modeling stage of intelligent models, and proposes a sensitivity analysis method based on Bayesian networks to establish a quantitative description model of the impact of defect factors on intelligent system failures. The overall research idea of this invention is as follows Figure 1 As shown below. The following example is given using the cat and dog recognition dataset from the Kaggle website as the object, as follows:
[0038] Step 1: Get the existing dataset Y to be used, and determine the N sample defect factors of interest, and discretize the N defect factors respectively, as follows: Figure 2 As shown, the following steps are included:
[0039] Step 1.1: Select one of the discretization methods, including equidistance, binarization, and K-means, to discretize the original data;
[0040] Step 1.2: Use the discretized data to build a Bayesian network model;
[0041] Step 1.3: Use the Bayesian network model to reason and analyze the causal relationship between variables within the system;
[0042] Step 1.4: Determine whether the pre-set inference accuracy is met. If so, return to step 1.1 and increase the discretization state value by 1. If not, output the maximum number of discretization states.
[0043] [Example] Take the cat and dog recognition dataset on the Kaggle website as an example. The dataset contains 25,000 pictures, 12,500 each of cats and dogs. Figure 3 The classification intelligent model used is a convolutional neural network. The possible defect factors in the construction process of the intelligent model data sample are analyzed as shown in Table 1.
[0044] Table 1 Clues to intelligent model defect factors
[0045]
[0046] Data balance, sample size, and feature engineering steps such as data normalization and image enhancement in the sample construction process are selected as sample defect factors. These four factors are discretized, data normalization and image enhancement are binarized, and the state number of data balance and sample size is taken as 10.
[0047] Step 2: Determine the evaluation indicators of the model, establish a Bayesian network between sample defects and evaluation indicators, and express sample defects as a Bayesian network structure and parameter constraints, including normative constraints, interval constraints, intra-distribution constraints, and inter-distribution constraints, as follows:
[0048] The normative constraint means that the sum of all parameters when the parent node has the same value is 1, as shown in formula (6):
[0049]
[0050] θ jk Indicates the probability value of the kth child node of the jth parent node in the network.
[0051] Interval constraints indicate that the parameters belong to a given interval, as shown in formula (7):
[0052] 0≤α jk ≤θ jk ≤β jk ≤1 (7)
[0053] α jk and β jk Represents the child node θ jk The probability of is bounded.
[0054] The intra-distribution constraint represents the size relationship between parameters in the same distribution when the parent nodes are the same, as shown in formula (8):
[0055]
[0056] θ jk and θ jk' They represent the probability values of two different child nodes under the same parent node.
[0057] The inter-distribution constraint represents the relationship between the parameters of different distributions when the probability of the child nodes is the same, as shown in formula (9):
[0058]
[0059] ν j'k and ν jk Indicates the probability of two different parent nodes having the same child node value.
[0060] [Example] Select the accuracy of the model in the test set as the evaluation indicator, and construct a Bayesian network of defect factors and accuracy, such as Figure 4 As shown, the corresponding constraints are specified.
[0061] Step 3: Perform sensitivity analysis on defects based on Bayesian network. The specific steps are as follows:
[0062] Step 3.1: Define the response function of each defect and evaluation index F(x) = g(X), X = x1, x2, ..., x n , define the response function to the defect factor x i The partial derivative of the response function to the defect factor x i Sensitivity S i , as shown in formula (10).
[0063]
[0064] When S i > 0, F(x) changes with x i increases with the increase of S i <0, F(x) increases with x i decreases with the increase of S i = 0, F(x) does not change with x i S i The absolute value reflects the change of F(x) with x. i The speed of change, that is, the function F(x) to the defect factor x i The sensitivity of each defect factor x can be obtained based on the conditional probability change degree of the Bayesian network. i The contribution of the value change to the change of the response function F(x).
[0065] Step 3.2: Determine the variation range of each defect factor, and select the i-th defect factor to construct the j-th sample data set Y according to the determined number of discretized states. ij .
[0066] Step 3.3: Input the constructed dataset into the intelligent model until the model converges, and use the same test set to test and obtain the dataset Y ij Evaluation index F ij .
[0067] Repeat steps 3.2-3.3 until all defect sample data sets have been tested.
[0068] Step 3.4: Input all the results into the Bayesian network, calculate the sensitivity index of each defect factor, and determine the contribution of each defect factor to the change of the model evaluation index.
[0069] [Example] A model failure is defined as a failure when its accuracy is less than 70%. Based on the number of discretized states for each defect factor, the sample size ranges from 1,000 to 10,000, the data balance ratio ranges from 1:9 to 9:1, and data augmentation and normalization are performed or not performed. Different datasets are constructed and fed into the constructed intelligent model for training and testing. The test results are fed into the Bayesian network to obtain the failure probability, sensitivity, and conditional failure probability of each defect factor, as shown in Table 2.
[0070] Table 2 Test results
[0071]
[0072] It can be seen that for the intelligent model in this case, the sensitivity of the defect factors follows the same order as the conditional failure probability. In this case, the sample size defect factor has the highest conditional failure probability, representing the weakest link in the system and the key factor affecting intelligent system failure.
Claims
1. A failure mechanism analysis method for intelligent system development modeling based on Bayesian network, Features The following steps are involved: Step 1: Obtain the existing data set Y to be used, determine the N sample defect factors of interest, and discretize the N defect factors respectively, including the following steps: Step 1.1: Select a discretization method among equal spacing, binarization and K-means to discretize the original data; Step 1.2: Use the discretized data to build a Bayesian network model; Step 1.3: Use the Bayesian network model to reason and analyze the cause-effect relationship of each variable within the system; Step 1.4: Determine whether the preset inference accuracy is met. If so, return to step 1.1 and increase the discretization state value by 1. If not, output the maximum number of discretization states. Step 2: Determine the evaluation index of the model, establish a Bayesian network between sample defects and evaluation indexes, and express sample defects as Bayesian network structure and parameter constraints, including normative constraints, interval constraints, intra-distribution constraints, and inter-distribution constraints, as follows: (1) The normative constraint means that when the parent node probabilities have the same value, the sum of all parameters is 1, as shown in the following formula: Among them, θ jk Indicates the probability value of the kth child node of the jth parent node in the network; (2) Interval constraints indicate that the parameter belongs to a given interval, as shown below: 0≤α jk ≤θ jk ≤β jk ≤1 Among them, α jk and β jk Represents the child node θ jk The probability of taking upper and lower bounds; (3) The intra-distribution constraint represents the size relationship between parameters in the same distribution when the parent nodes are the same, as shown in the following formula: Among them, θ jk and θ jk' They represent the probability values of two different child nodes under the same parent node; (4) The inter-distribution constraint indicates the relationship between the parameters of different distributions when the probability values of the child nodes are the same, as shown in the following formula: Among them, ν j'k and ν jk Indicates the probability of two different parent nodes taking the same child node value; Step 3: Perform sensitivity analysis on defects based on Bayesian network. The specific steps are as follows: Step 3.1: Define the response function of each defect and evaluation index F(x) = g(X), X = x 1 ,x 2 ,...,x n , define the response function to the defect factor x i The partial derivative of the response function to the defect factor x i Sensitivity S i , as follows: When S i > 0, F(x) increases with x i increases with the increase of S i <0, F(x) increases with x i decreases with the increase of S i = 0, F(x) does not change with x i changes with the changes of S i The absolute value reflects the change of F(x) with x. i The speed of change, that is, the function F(x) to the defect factor x i The sensitivity of each defect factor x can be obtained based on the conditional probability change degree of the Bayesian network. i The contribution of the value change to the change of the response function F(x); Step 3.2: Determine the variation range of each defect factor, and select the i-th defect factor to construct the j-th sample data set Y according to the determined number of discretized states. ij ; Step 3.3: Input the constructed data set into the intelligent model until the model converges, and use the same test set to test and obtain the data set Y ij The evaluation index F ij ; Repeat steps 3.2-3.3 until all defect sample data sets are tested; Step 3.4: Input all the results into the Bayesian network, calculate the sensitivity index of each defect factor, and determine the contribution of each defect factor to the change of the model evaluation index.
