Robustness detection method and device of intelligent decision-making system, electronic equipment and medium

Abstract

The invention provides a robustness detection method and device of an intelligent decision-making system, electronic equipment and a medium, and relates to the technical field of artificial intelligence detection, and the method comprises the steps: processing original data in an original data set, and generating a detection data set; the detection data set comprises transformation data obtained by processing original data, antagonism data and optimization data obtained based on a fuzzy algorithm; inputting the detection data set into an intelligent decision-making system, and processing data in the detection data set based on the intelligent decision-making system; obtaining a neuron coverage rate, a network layer coverage rate and a neuron pair coverage rate of the intelligent decision-making system in the processing process; and determining the robustness of the intelligent decision-making system based on the neuron coverage rate, the network layer coverage rate and the neuron pair coverage rate. The method is used for accurately determining the robustness of the intelligent decision-making system.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence detection technology, and in particular to a robustness detection method, device, electronic device and medium for an intelligent decision-making system. Background Art

[0002] Intelligent decision-making systems can be applied in many fields to help users make decisions. Robustness testing is a means to test the stability and reliability of intelligent decision-making systems.

[0003] At present, a method for testing the robustness of an intelligent decision-making system is to generate test cases by performing linear transformation, affine transformation and other processing on the original data, and then process the test cases through an intelligent decision-making system. The accuracy of the processing results represents the robustness of the intelligent decision-making system, but it cannot guarantee the accurate determination of the robustness of the intelligent decision-making system. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a robustness detection method, device, electronic device and medium for an intelligent decision-making system.

[0005] In a first aspect, the present disclosure provides a method for detecting robustness of an intelligent decision-making system, comprising:

[0006] Processing the original data in the original data set to generate a detection data set; the detection data set includes: transformed data obtained by processing the original data, adversarial data, and optimized data obtained based on a fuzzy algorithm;

[0007] Inputting the detection data set into the intelligent decision-making system, and processing the data in the detection data set based on the intelligent decision-making system;

[0008] Obtaining the neuron coverage, network layer coverage, and neuron pair coverage of the intelligent decision-making system during the processing;

[0009] The robustness of the intelligent decision-making system is determined based on the neuron coverage, the network layer coverage, and the neuron pair coverage.

[0010] In some embodiments, the processing of the raw data in the raw data set to generate the detection data set includes:

[0011] Performing linear transformation and / or affine transformation on the original data in the original data set to obtain the transformed data;

[0012] Inputting the original data into the trained generative adversarial network model for processing, obtaining and outputting the adversarial data;

[0013] The original data is processed using multi-coverage guided fuzz testing to obtain the optimized data.

[0014] In some embodiments, the neuron coverage rate of the intelligent decision-making system in the acquisition process includes:

[0015] For each neuron in the intelligent decision-making system, determining activation state data of the neuron according to a relationship between an output value of the neuron and an activation threshold;

[0016] The neuron coverage of the intelligent decision-making system is calculated based on the activation state data of the neurons.

[0017] In some embodiments, the step of calculating the neuron coverage of the intelligent decision-making system based on the activation state data of the neurons includes:

[0018] Calculating the activated neuron coverage rate, k-node neuron coverage rate, and neuron boundary coverage rate of the intelligent decision-making system according to the activation state data of the neurons;

[0019] The neuron coverage rate of the intelligent decision-making system is obtained by weighted summing the activated neuron coverage rate, the k-node neuron coverage rate, and the neuron boundary coverage rate.

[0020] In some embodiments, the network layer coverage of the intelligent decision-making system in the acquisition process includes:

[0021] For each network layer in the intelligent decision-making system, the output value of each neuron in the network layer is obtained, and the proportion of neurons whose output values ​​rank in the top K positions in all neurons in the network layer is determined as the coverage rate of the top K positions of neurons corresponding to the network layer;

[0022] The network layer coverage rate in the intelligent decision-making system is obtained by weighted summing up the coverage rates of the first K neurons in each network layer.

[0023] In some embodiments, the neuron pair coverage of the intelligent decision-making system in the acquisition process includes:

[0024] For each neuron pair in the intelligent decision-making system, determining the output value, sign, and distance of two neurons in the neuron pair;

[0025] The neuron pair coverage in the intelligent decision-making system is determined according to the output values, signs, and distances of the two neurons in each neuron pair.

[0026] In some embodiments, determining the neuron pair coverage in the intelligent decision-making system according to the output values, signs, and distances of the two neurons in each neuron pair includes:

[0027] For each neuron pair, if the signs of two neurons in the neuron pair are the same, it is determined that the neuron pair is covered; or,

[0028] If the signs of the two neurons in the neuron pair are the same and the difference between the output values ​​of the two neurons is greater than a first threshold, it is determined that the neuron pair is covered; or,

[0029] If the signs of the two neurons in the neuron pair are the same, and the output values ​​of the two neurons are both greater than the second threshold, it is determined that the neuron pair is covered; or,

[0030] If the difference between the output values ​​of two neurons in the neuron pair is greater than a first threshold, and the output values ​​of the two neurons are both greater than a second threshold, it is determined that the neuron pair is covered;

[0031] The proportion of covered neuron pairs in the neuron pairs of the intelligent decision-making system is calculated to determine the coverage rate of neuron pairs in the intelligent decision-making system.

[0032] In a second aspect, the present disclosure provides a robustness detection device for an intelligent decision-making system, comprising:

[0033] The original data processing module is used to process the original data in the original data set to generate a detection data set; the detection data set includes: the transformed data obtained by processing the original data, the adversarial data, and the optimized data obtained based on the fuzzy algorithm;

[0034] A detection data processing module, used for inputting the detection data set into the intelligent decision-making system, and processing the data in the detection data set based on the intelligent decision-making system;

[0035] A coverage calculation module, used to calculate the neuron coverage, network layer coverage, and neuron pair coverage of the intelligent decision-making system during the processing;

[0036] A robustness evaluation module is used to determine the robustness of the intelligent decision-making system based on the neuron coverage, the network layer coverage, and the neuron pair coverage.

[0037] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the robustness detection method of the intelligent decision-making system described in the first aspect or any optional implementation manner of the first aspect when calling the computer program.

[0038] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robustness detection method for the intelligent decision-making system described in the first aspect or any optional implementation manner of the first aspect.

[0039] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0040] The robustness detection method of the intelligent decision-making system provided by the embodiment of the present disclosure includes: processing the original data in the original data set to generate a detection data set; the detection data set includes: transformation data obtained by processing the original data, adversarial data, and optimization data obtained based on the fuzzy algorithm; inputting the detection data set into the intelligent decision-making system, and processing the data in the detection data set based on the intelligent decision-making system; obtaining the neuron coverage, network layer coverage, and neuron pair coverage of the intelligent decision-making system during the processing; and determining the robustness of the intelligent decision-making system based on the neuron coverage, network layer coverage, and neuron pair coverage. On the one hand, the embodiment of the present disclosure processes the original data set through a variety of data processing methods to obtain a variety of detection data sets for the robustness detection of the intelligent decision-making system, which can improve the comprehensiveness of the robustness detection. On the other hand, the embodiment of the present disclosure determines the robustness of the intelligent decision-making system by evaluating indicators based on multiple dimensions such as neuron coverage, network layer coverage, and neuron pair coverage, which can improve the reliability of the evaluation results. The diverse detection data and multi-dimensional evaluation indicators improve the accuracy of determining the robustness of the intelligent decision-making system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flowchart of a method for detecting robustness of an intelligent decision-making system provided by an embodiment of the present disclosure;

[0044] Figure 2 A schematic diagram of a generative adversarial network model for generating adversarial data provided by one embodiment of the present disclosure;

[0045] Figure 3A schematic diagram of an ontology model for generating detection data based on multi-coverage guided fuzz testing provided by an embodiment of the present disclosure;

[0046] Figure 4 A schematic diagram of a process for generating detection data based on multi-coverage guided fuzz testing provided by an embodiment of the present disclosure;

[0047] Figure 5 A structural block diagram of a robustness detection device for an intelligent decision-making system provided by an embodiment of the present disclosure;

[0048] Figure 6 An internal structure diagram of an electronic device provided for one embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to make the purpose, implementation mode and advantages of the present disclosure clearer, the exemplary implementation mode of the present disclosure will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present disclosure. Obviously, the described exemplary embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments.

[0050] Based on the exemplary embodiments described in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to the present disclosure. In addition, although the disclosure in the present disclosure is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately. It should be noted that the brief description of the terms in the present disclosure is only for the convenience of understanding the implementation methods described below, and is not intended to limit the implementation methods of the present disclosure. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0051] For intelligent decision-making systems, robustness testing can effectively ensure their quality and reliability in safety-critical issues. At present, the robustness testing methods for intelligent decision-making systems are mainly focused on the generation of test cases. Most of them use linear transformation, affine transformation and other methods to perform simple transformations on the original data to generate test cases, and use accuracy to represent robustness. The current testing methods cannot check the capabilities of intelligent decision-making systems in multiple dimensions, and lack accurate evaluation of the robustness of intelligent decision-making systems.

[0052] Based on this, the embodiments of the present disclosure provide a method for detecting the robustness of an intelligent decision-making system, which generates detection data in a variety of ways to ensure the diversity of detection data, and determines the robustness of the intelligent decision-making system through multi-dimensional evaluation indicators such as neuron coverage, network layer coverage, and neuron pair coverage during the detection data processing. The diversity of detection data and the multi-dimensionality of evaluation indicators ensure accurate evaluation of the robustness of the intelligent decision-making system.

[0053] The robustness detection method of the intelligent decision-making system provided by the present disclosure will be described and elaborated through specific exemplary embodiments below.

[0054] Figure 1 A flowchart of a robustness detection method for an intelligent decision-making system provided by an embodiment of the present disclosure, referring to Figure 1 As shown, the robustness detection method of the intelligent decision-making system provided in this embodiment includes the following steps S11 to S13.

[0055] S11. Process the original data in the original data set to generate a detection data set.

[0056] The detection data set includes: transformed data obtained by transforming the original data, adversarial data, and optimized data obtained based on a fuzzy algorithm. The original data includes training data for training the intelligent decision-making system, and may also include other data of the same type as the training data. The transformed data includes data obtained by performing a simple transformation on the original data, such as data obtained by performing a linear transformation, an affine transformation, etc. on the original data. Adversarial data can be understood as adversarial samples, which can cause the neural network to make incorrect predictions with high confidence. The optimized data is data obtained by processing the original data based on a fuzzy algorithm.

[0057] Specifically, the transformed data, adversarial data, and optimized data obtained by processing the original data can constitute a detection data set. Of course, the detection data set can also include other data, which is not limited in the embodiments of the present disclosure.

[0058] Exemplarily, each test data in the test data set can be obtained by: performing linear transformation and / or affine transformation on the original data to obtain transformed data; inputting the original data into a trained generative adversarial network for processing to obtain and output adversarial data; using multi-coverage guided fuzzy testing to process the original data to obtain optimized data. The transformed data may include data obtained based on linear transformation, data obtained based on affine transformation, or data obtained based on linear transformation and affine transformation.

[0059] In some embodiments, in order to generate an appropriate amount of deceptive adversarial data, a generative adversarial network is constructed, and the original data is input into the generative adversarial network to generate corresponding observation data, thereby achieving the purpose of deceiving the discriminant network and increasing the detection coverage of the neural network in the intelligent decision-making system. The generative adversarial network can be a network module of the intelligent decision-making system.

[0060] Combination Figure 2 As shown, Figure 2 Schematic diagram of the generative adversarial network model for generating adversarial data, see Figure 2As shown, the model for generating adversarial data can be a generative adversarial network (GAN), which is a deep learning model that includes a generative model (G) and a discriminative model (D). The original data is input into the generative adversarial network as sample data. The generative model captures the distribution of the sample data and simulates the distribution of the real input sample. Its input is high-dimensional random noise z, and its output is a false sample. The discriminator D detects whether the current sample is a real sample, and its input is a real sample and a false sample. Among them, the discriminative model is a binary classifier used to determine whether the input data is real.

[0061] The disclosed embodiment also includes an optimization process for a generative adversarial network, which first includes an optimization of an objective function of the generative adversarial network, and the objective function is as follows: Formula (1).

[0062]

[0063] Among them, P data represents the distribution of real data, P z Denotes the distribution of the original noise, and E refers to the expected value. D is obtained by first fixing G to maximize V(D, G), then fixing D and minimizing V(D, G) to obtain G, and so on iteratively until the entire model reaches the predetermined goal.

[0064] The disclosed embodiment uses the original data as a whole to train a deep convolutional generative adversarial network G1 to simulate the distribution of the original sample set; secondly, in the black-box attack scenario, the target model is black-box copied by the model distillation method to obtain a local copy of the target model; then the output of G1 is used as input, and the distilled model is used as the target model to train the generative adversarial network G2. In the case of a targeted attack, the target category needs to be input, and G2 is used to generate perturbations of the input data for the target category; finally, the sample and the perturbation are added and normalized with a pixel gray value interval to obtain an adversarial sample.

[0065] Among them, the construction of the adversarial generative network may include the following process: using the InfoGAN network to interpret the input signal of the generator, and achieving corresponding changes by controlling the corresponding dimensional variables. Since the traditional GAN ​​network has almost no constraints on the generator, when there are many sample categories, it becomes extremely difficult to obtain samples of a specific category because the generated content of the generator is too free. Therefore, by adding constraints to the generator, the generator can be guided to generate samples of the desired category. InfoGAN divides the input signal z into two parts, one is random noise z, and the other is an interpretable signal c with implicit meaning, which serves as the input of the generator G.

[0066] By using mutual information to constrain, z and G(z,c) are highly correlated. Since the mutual information term λI(c; G(z,c)) is difficult to maximize directly, it is necessary to first obtain the posterior probability term P(c|x). To solve this problem, an auxiliary definition of the infimum of Q(c|x) is introduced to approximate P(c|x). Therefore, the objective function of InfoGAN is redefined as shown in formula (2), where λ is a hyperparameter. On this basis, a sample generator G1 based on transposed convolution is constructed.

[0067] min G max D V InfoGAN (D,G,Q)=V(D,G)-λL1(G,Q) formula (2);

[0068] The input of the perturbation generator G2 is a sample and the target of the attack, and the output is the noise corresponding to the sample and aimed at the desired attack target. The adversarial sample is obtained by adding the sample to the noise output by G2 and then normalizing it, which can be used to perform semi-white box and black box attacks on the target model.

[0069] In this process, in order to ensure the authenticity of the generated adversarial samples (adversarial data), which will not be too distorted compared with the initial samples, and to reduce the noise as much as possible under the limitation of the perturbation coefficient, the discriminator loss function L GAN The definition is as shown in the following formula (3):

[0070] L GAN =E x logD(x)+E x log(1-D(G(x)+x)) formula (3);

[0071] The target model f is used to verify the attack effect of the current adversarial sample on the target model and minimize the target model loss function L adv It can make the result of adversarial samples in the attack process closer to the expected value. adv The calculation of is shown in formula (4):

[0072] L adv =E x l f (x+G(x), t) formula (4);

[0073] Among them, l f represents the loss function of the target model f, whose input is the constructed adversarial sample. In a targeted attack, t is the attack target, minimizing L adv It can make the result of adversarial samples in the attack process closer to the expected category; in untargeted attacks, t is the true category of the sample, maximizing L advThe result of the adversarial sample in the attack process can be made infinitely far away from the real result, realizing the non-targeted attack of the adversarial sample. GAN and L adv The model parameters of G2 are modified through back propagation, so that G2 can generate adversarial samples that are as consistent as possible with the smallest possible noise cost.

[0074] In order to enhance the portability of adversarial samples, the embodiments of the present disclosure can also use model distillation to perform black-box attacks. Specifically, model distillation is used to obtain a local copy of the target model, and then the distilled model is attacked using the same attack method as the white-box attack. First, a distillation model f of the target black-box model b is constructed according to the experimental scenario. Models b and f can minimize the objective function in the model distillation process. The objective function is expressed as shown in formula (5):

[0075] V=arg min x E x H(f(x), b(x)) formula (5);

[0076] Among them, f(x) and b(x) represent the output results of the same image in the distillation model and the black box model respectively, and H represents the cross entropy loss. Minimizing this function makes the output results of model f gradually approach model b, and some key weight vectors of the two models can gradually match. Compared with the traditional black box attack method that uses the transferability of adversarial samples, the distillation model will significantly improve the attack success rate under the black box attack method.

[0077] In some embodiments, the original data set is used as the test data set for input, and the original data is processed using multi-coverage guided fuzz testing to obtain optimized data. Figure 3 As shown, Figure 3 A schematic diagram of an ontology model for generating detection data based on multi-coverage guided fuzz testing provided for an embodiment of the present disclosure. Fuzz testing is an automatic software detection technology based on defect injection, which discovers system defects by providing unexpected inputs to the target system or software and monitoring abnormal results. Its effectiveness depends largely on the code coverage of unexpected inputs in the target program. Fuzz testing can be regarded as a process of searching for a high-quality detection data set to obtain higher code coverage and as many anomalies as possible.

[0078] Traditional coverage-guided fuzz testing randomly mutates the system input to detect edge cases in the software, and can only detect shallow errors, such as format parsing errors. However, most DNN systems require input to be in a special format, and traditional coverage-guided fuzz testing methods are not applicable. Based on this, the embodiments of the present disclosure provide an optimized fuzz testing algorithm. Compared with the original fuzzer, the calculation process of the optimized fuzz testing algorithm is as follows:

[0079] Combination Figure 4 As shown, Figure 4 A flow chart of generating test data based on multi-coverage guided fuzz testing provided for an embodiment of the present disclosure, wherein the original data set is input into a selector as a test data set, and a test data is selected from the constructed test data set using the selector, and is passed to the fuzzifier for mutation, such as adding white noise to the image matrix to obtain mutated data. Then, the mutated data (test case) is input into the neural network for training, and the output result is divided into two cases. If the output result satisfies the objective function, that is, a crash is triggered or an anomaly is detected, it means that the fuzz test algorithm has detected a defect in the neural network; if the output result does not meet the objective function, the coverage analyzer is used to determine whether the new code is covered, and further confirm whether to update the test data set.

[0080] For the optimized fuzzer, we first need to understand the specifications and protocols of the input data. At the beginning, the detection data is generated by the random number generator and the incremental controller, and then input into the neural network for execution. At this time, there are also two cases. If the output result meets the objective function, that is, a crash is triggered or an anomaly is detected, the anomaly is recorded; if the output result does not meet the objective function, the incremental controller will dynamically adjust the data change range, generate a new set of detection data, and conduct the next round of detection. In actual detection, the parameters of the incremental controller will be changed many times to make the generated detection data more evenly distributed.

[0081] The optimized fuzzy algorithm has obvious advantages when detecting detection systems containing a large number of non-numerical defects. It can detect more defects per unit time, thereby further improving code coverage. However, this efficiency improvement still has limitations and is only suitable for detecting non-numerical defects in the system to be tested. When the defects are of other types, the optimization algorithm cannot show its advantages. Therefore, based on the optimized fuzzy test, the multi-coverage guided fuzzer algorithm of the embodiment of the present disclosure starts from another perspective of the fuzzy testing method and uses a generation-based fuzzy testing method. The specific process is: by trying to expand the random range, reduce the mutation period, and use the threshold (New Threshold Value, NTH) to screen new coverage, and by training the neural network to a certain extreme, it is likely to obtain the defects of the detection object under this condition.

[0082] Specifically, the initial seed raw data is first preprocessed to generate a test data set. A test data is selected from the generated test data set through a selector and passed to the fuzzifier for mutation. For example, noise is added to the image data to obtain the mutated data. The mutated data is then input into the neural network for training. At this time, the output results are divided into two types. If the output result satisfies the objective function, that is, a crash is triggered or an anomaly is detected, it means that the fuzz test algorithm has detected a defect in the neural network. Otherwise, the coverage analyzer is used to determine whether it covers the new coverage unit, and further confirm whether to update the test data set. This achieves the optimization of the original data set.

[0083] The disclosed embodiments provide a deep learning fuzz testing method based on multi-coverage guidance, so that the multi-coverage guided fuzz testing method generates detection data, which can improve the pass rate of detection data during the verification period, cover code segments that may be high-risk, and discover hidden defects. At the same time, it guides the fuzz testing of the intelligent decision-making system through multiple coverage standards, which is conducive to the detection of deep defects.

[0084] In the disclosed embodiment, transformed data corresponding to the original data is obtained through linear transformation and affine transformation, adversarial data is generated through a generative adversarial network, and optimized data is obtained based on a fuzzy algorithm. These four data processing methods not only increase the complexity of generating a detection data set, but also increase the randomness of the generation, thereby generating more diverse detection data.

[0085] S12, inputting the detection data set into the intelligent decision-making system, and processing the data in the detection data set based on the intelligent decision-making system.

[0086] Among them, the intelligent decision-making system includes the structural particularity of the neural network model. The formation of the function and parameter structure of the neural network model is closely related to the data, which leads to certain uncertainties in the formation of the functional structure, model structure, and logical structure of the neural network. In order to achieve the adequacy of the detection coverage of the neural network, the embodiment of the present disclosure detects the coverage of the neural network from multiple levels such as the neuron level, the neural network layer level, and the neuron pair layer level.

[0087] Specifically, the model characteristics of the corresponding neural network structure in the intelligent decision-making system are first analyzed; secondly, according to the analysis results of the corresponding coverage method at different coverage levels, the corresponding detection coverage method is determined for detection; finally, the selection criteria of the neural network detection coverage method are determined, the neural network coverage range judgment criteria are established, and its detection coverage is analyzed.

[0088] S13. Obtain the neuron coverage, network layer coverage, and neuron pair coverage of the intelligent decision-making system during the processing.

[0089] Specifically, we first establish a detection method for neuron coverage, network layer coverage, and neuron pair coverage. In the process of using the intelligent decision-making system to process the detection data set, we calculate the neuron coverage, network layer coverage, and neuron pair coverage of the intelligent decision-making system to evaluate the adequacy of the detection data set in covering the neural network in the intelligent decision-making system.

[0090] Exemplarily, the method for calculating neuron coverage may include: for each neuron in the intelligent decision-making system, determining the activation state data of the neuron according to the relationship between the output value of the neuron and the activation threshold; and calculating the neuron coverage of the intelligent decision-making system based on the activation state data of the neuron.

[0091] The activation status data includes an activated state and an unactivated state.

[0092] Specifically, for the neuron activation threshold, it is determined whether the output value of the neuron in the neural network is greater than the activation threshold. When it is greater than the activation threshold, the neuron is activated and is in an activated state; otherwise, the neuron is activated and is in an inactivated state. Then, the coverage rate of neurons in the activated state is calculated to realize the calculation of the neuron coverage rate of the intelligent decision-making system based on the neuron activation state data.

[0093] When calculating the neuron coverage of the intelligent decision-making system based on the activation state data of the neurons, a variety of neuron activation coverage detection methods can be used, for example, detecting activation neuron coverage; or detecting k-node neuron coverage; or detecting neuron boundary coverage, etc. That is, the neuron coverage of the intelligent decision-making system can be determined by the neuron coverage, k-node neuron coverage, and / or neuron boundary coverage.

[0094] In some embodiments, the process of determining the neuron coverage of the intelligent decision-making system includes: calculating the activated neuron coverage, k-node neuron coverage, and neuron boundary coverage of the intelligent decision-making system based on the activation state data of the neurons; and obtaining the neuron coverage of the intelligent decision-making system by weighted summation of the activated neuron coverage, k-node neuron coverage, and neuron boundary coverage.

[0095] The more neurons that can be activated by the detection data set, the higher the neuron coverage rate. In addition to covering the main functional behaviors of the neural network, it is more likely to trigger the extreme behaviors of the neural network. Therefore, the coverage rate of activated neurons can be obtained by calculating the ratio of the number of activated neurons to the total number of neurons. In order to improve the coverage rate of activated neurons, the original data can be modified so that the newly generated detection data can activate more neurons, or different neurons can be activated through data conversion operations to increase the neuron activation rate.

[0096] The k-section neuron coverage rate is the neuron coverage rate obtained based on the k-section neuron coverage test method, which is used to measure the input test data set to cover the neuron activation value range [low n ,high n ] to a certain extent.

[0097] Specifically, the neuron activation value interval [low n ,high n ] is divided into k parts, using represents the set of values ​​in section i, where 1≤i≤k. If Indicates that the i-th section is covered by input x. There is a set of inputs T and a neuron n, and the coverage is defined as the ratio of the number of sections covered by T to the headquarters score. The k-section neuron coverage of neuron n is That is, the detection set T covers some of the k segments. Therefore, the coverage rate of the k-segment neurons of the neural network is the sum of the coverage rates of N neurons, as shown in formula (6).

[0098]

[0099] Neuron boundary coverage is the neuron coverage obtained based on the neuron boundary coverage test, which aims to test the extreme behavior of the neural network, that is, the performance when the activation value of the neuron exceeds its normal training range.

[0100] Combined with the above example, due to the existence of some detection samples, the output of the neural network does not fall into the expected output domain, but falls into [low n ,high n ] outside. Available (-∞, low n )∪(high n ,+∞) describes the extreme behavior of the neural network. Therefore, it will fall into (high n ,+∞) area is defined as will fall into (-∞,low n The neuronal ensemble in the region is defined as

[0101] In some embodiments, the neuron boundary coverage rate can be obtained by calculating the ratio of the number of neurons falling into the two regions to twice the total number of neurons, as shown in the following formula (7).

[0102] Cov B =(|N UPPER |+|N LOWER |) / 2×|N|Formula (7);

[0103] Since neurons that are activated and cause extreme behaviors have a greater impact on the learning effect of the neural network, enhanced neuron activation coverage can also be used. Therefore, in some embodiments, the high n ,+∞) to the total number of neurons to describe the coverage of activated neurons for extreme behaviors, that is, the neuron boundary coverage rate can also be obtained by the following formula (8).

[0104] Cov SNA =|N UPPER | / |N|Formula (8);

[0105] In a neural network, the neurons that determine the main functional behavior of the network system are often relatively active neurons. Therefore, the detection data set should cover as many neurons with high activity as possible. In some embodiments of the present disclosure, the network layer coverage is calculated from the perspective of the network layer.

[0106] In some embodiments, the network layer coverage rate can be obtained through the following process: for each network layer in the intelligent decision-making system, the output value of each neuron in the network layer is obtained, and the proportion of neurons whose output values ​​are ranked in the top K positions in all neurons in the network layer is determined as the top K position neuron coverage rate corresponding to the network layer; and the network layer coverage rate in the intelligent decision-making system is obtained by taking a weighted sum of the top K position neuron coverage rates of each network layer.

[0107] The network layer coverage can be understood as the coverage obtained based on the top-k neuron coverage method, which is used to describe the coverage of the top k neurons in the neural network. In the disclosed embodiment, the more active neurons in each neural network layer are separated, and the detection coverage in the neural network layer is calculated, thereby realizing the coverage analysis of the entire network structure.

[0108] Specifically, for each network layer in the intelligent decision-making system, for a given detection input x and neurons n1 and n2 on the same layer, if φ(x,n1)>φ(x,n2), it means that n1 is more active than the given n2, that is, the neurons in the same layer have larger outputs and are more active. k (x,i) represents the k neurons with the largest values ​​in the i-th layer given an x. The top-k neuron coverage measures the number of the most active k neurons in each layer. The definition of the top-k neuron coverage is shown in formula (9).

[0109]

[0110] Given a detection input x, the top-k neuron sequence of each layer also forms a pattern: TKNPat(T,k)=|{(top k(x,1),…,top k (x,l))|x∈T}|, that is, the top-k neuron patterns represent different activation scenarios of the top hyperactive neurons in each layer.

[0111] The coverage detection method based on neuron pairs can obtain the neuron pair coverage rate.

[0112] Specifically, a neuron pair includes two neurons with a connection relationship between adjacent layers of a neural network. The neurons in the l-1 layer are used as conditions, and the neurons in the l layer are used as decisions. Combined with the sign, size and other parameters of the output value of the neurons, the neuron pairs are planned from three directions: neuron sign change, value change, and distance change. The coverage of the neuron pairs can be obtained.

[0113] In some embodiments, for each neuron pair in the intelligent decision-making system, the output value, sign, and distance of the two neurons in the neuron pair are determined; based on the output value, sign, and distance of the two neurons in each neuron pair, the coverage of the neuron pairs in the intelligent decision-making system is determined. The change in the sign of the neuron means that if the input and output of the neuron are equal, the sign of the neuron is positive; otherwise, it is negative. For example, if but otherwise Given a neuron Input detection cases x1 and x2, so that Then the neuron The sign of has changed and is expressed as On the contrary, it is expressed as

[0114] Given a neuron If the detection data changes from x1 to x2, the neuron The input value of (absolute change) or (relative change) when the value changes At the same time, the neuron symbol remains unchanged, i.e. Then the neuron A value change occurs, indicating This is because when the sign of the neuron does not change, the change of the neuron's input will cause the neuron's output value to change, and the final output of the neural network will also be affected, which may cover extreme behaviors.

[0115] Distance refers to the difference between the output values ​​of neurons. For example, given the l-th layer of neurons, if the detection data changes from x1 to x2, the distance between the l-th layer of neurons satisfies Lp norm, that is, when When the distance changes And the symbols of all neurons in the lth layer remain unchanged, that is, It means that the distance of the decision neuron in layer l has changed, which is expressed as dc(h,l,x1,x2); otherwise, it is expressed as

[0116] In some embodiments, the coverage rate of neuron pairs in the intelligent decision-making system is determined based on the output values, signs, and distances of the two neurons in each neuron pair, including: for each neuron pair, if the signs of the two neurons in the neuron pair are the same, the neuron pair is determined to be covered; or, if the signs of the two neurons in the neuron pair are the same and the difference between the output values ​​of the two neurons is greater than a first threshold, the neuron pair is determined to be covered; or, if the signs of the two neurons in the neuron pair are the same and the output values ​​of the two neurons are both greater than a second threshold, the neuron pair is determined to be covered; or, if the difference between the output values ​​of the two neurons in the neuron pair is greater than a first threshold and the output values ​​of the two neurons are both greater than a second threshold, the neuron pair is determined to be covered; calculate the proportion of covered neuron pairs in the neuron pairs of the intelligent decision-making system to determine the coverage rate of neuron pairs in the intelligent decision-making system.

[0117] The same sign includes: the signs of neurons are all positive, or all negative, or all zero. According to the above definition of the influence of neurons on the determination of neurons, given neurons on ( is the conditional neuron, To determine the neuron) and the detection data x1, x2, there are four neuron pair coverage criteria based on this: symbol-symbol coverage, distance-symbol coverage, symbol-value coverage, and distance-value coverage. The neuron pair coverage rate is expressed as the ratio of the number of covered neuron pairs to the total number of neuron pairs.

[0118] S14. Determine the robustness of the intelligent decision-making system based on neuron coverage, network layer coverage, and neuron pair coverage.

[0119] Among them, the robustness of the intelligent decision-making system is positively correlated with the neuron coverage rate, network layer coverage rate, and neuron pair coverage rate. The larger the neuron coverage rate, network layer coverage rate, and neuron pair coverage rate, the better the robustness of the intelligent decision-making system.

[0120] Specifically, a total coverage rate can be determined based on the neuron coverage rate, the network layer coverage rate, and the neuron pair coverage rate. For example, by assigning corresponding weights to the neuron coverage rate, the network layer coverage rate, and the neuron pair coverage rate, the total coverage rate is determined by weighted summation based on the weights, and then the robustness of the intelligent decision-making system is evaluated according to the total coverage rate.

[0121] In the embodiments of the present disclosure, by using multiple coverage rates such as neuron coverage rate, network layer coverage rate, and neuron pair coverage rate as indicators for evaluating the robustness of the intelligent decision-making system, it is helpful to improve the comprehensiveness and reliability of the evaluation results.

[0122] The embodiment of the present disclosure provides a method for detecting the robustness of an intelligent decision-making system, including: processing the original data in the original data set to generate a detection data set; the detection data set includes: transformation data obtained by processing the original data, adversarial data, and optimization data obtained based on a fuzzy algorithm; inputting the detection data set into the intelligent decision-making system, and processing the data in the detection data set based on the intelligent decision-making system; obtaining the neuron coverage, network layer coverage, and neuron pair coverage of the intelligent decision-making system during the processing; and determining the robustness of the intelligent decision-making system based on the neuron coverage, network layer coverage, and neuron pair coverage. On the one hand, the embodiment of the present disclosure processes the original data set through a variety of data processing methods to obtain a variety of detection data sets for the robustness detection of the intelligent decision-making system, which can improve the comprehensiveness of the robustness detection. On the other hand, the embodiment of the present disclosure determines the robustness of the intelligent decision-making system by evaluating indicators based on multiple dimensions such as neuron coverage, network layer coverage, and neuron pair coverage, which can improve the reliability of the evaluation results. The diverse detection data and multi-dimensional evaluation indicators improve the accuracy of determining the robustness of the intelligent decision-making system.

[0123] Based on the same inventive concept, as an implementation of the above method, the embodiment of the present disclosure also provides a robustness detection device for the intelligent decision-making system provided by the above embodiment. The device embodiment corresponds to the above method embodiment. For ease of reading, the device embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the robustness detection device for the intelligent decision-making system in this embodiment can correspond to all the contents of the above method embodiment.

[0124] Figure 5 A structural diagram of a robustness detection device for an intelligent decision-making system provided in an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the robustness detection device 500 of the intelligent decision-making system provided in this embodiment includes:

[0125] The original data processing module 510 is used to process the original data in the original data set to generate a detection data set; the detection data set includes: the transformed data obtained by processing the original data, the adversarial data, and the optimized data obtained based on the fuzzy algorithm;

[0126] The detection data processing module 520 is used to input the detection data set into the intelligent decision-making system and process the data in the detection data set based on the intelligent decision-making system;

[0127] A coverage calculation module 530 is used to obtain the neuron coverage, network layer coverage, and neuron pair coverage of the intelligent decision-making system during the processing;

[0128] The robustness evaluation module 540 is used to determine the robustness of the intelligent decision-making system based on neuron coverage, network layer coverage, and neuron pair coverage.

[0129] In some embodiments, the original data processing module 510 is specifically used to perform linear transformation and / or affine transformation on the original data in the original data set to obtain transformed data; input the original data into the trained generative adversarial network model for processing to obtain and output adversarial data; and use multi-coverage guided fuzzy testing to process the original data to obtain optimized data.

[0130] In some embodiments, the coverage calculation module 530 is specifically used to determine the activation state data of each neuron in the intelligent decision-making system according to the relationship between the output value of the neuron and the activation threshold; and calculate the neuron coverage of the intelligent decision-making system based on the activation state data of the neuron.

[0131] In some embodiments, the coverage calculation module 530 is specifically used to calculate the activated neuron coverage, k-node neuron coverage, and neuron boundary coverage of the intelligent decision-making system based on the activation state data of the neurons; the neuron coverage of the intelligent decision-making system is obtained by weighted summation of the activated neuron coverage, k-node neuron coverage, and neuron boundary coverage.

[0132] In some embodiments, the coverage calculation module 530 is specifically used to obtain the output value of each neuron in the network layer for each network layer in the intelligent decision-making system, and determine the proportion of neurons whose output values ​​​​rank in the top K positions in all neurons in the network layer as the coverage rate of the top K positions of neurons corresponding to the network layer; by taking the weighted sum of the coverage rates of the top K positions of neurons in each network layer, the network layer coverage rate in the intelligent decision-making system is obtained.

[0133] In some embodiments, the coverage calculation module 530 is specifically used to determine the output value, sign, and distance of the two neurons in each neuron pair in the intelligent decision-making system; and determine the coverage of the neuron pairs in the intelligent decision-making system based on the output value, sign, and distance of the two neurons in each neuron pair.

[0134] In some embodiments, the coverage calculation module 530 is specifically used to, for each neuron pair, determine that the neuron pair is covered if the signs of the two neurons in the neuron pair are the same; or, if the signs of the two neurons in the neuron pair are the same and the difference between the output values ​​of the two neurons is greater than a first threshold, determine that the neuron pair is covered; or, if the signs of the two neurons in the neuron pair are the same and the output values ​​of the two neurons are both greater than a second threshold, determine that the neuron pair is covered; or, if the difference between the output values ​​of the two neurons in the neuron pair is greater than a first threshold and the output values ​​of the two neurons are both greater than a second threshold, determine that the neuron pair is covered; calculate the proportion of covered neuron pairs in the neuron pairs of the intelligent decision-making system to determine the coverage rate of neuron pairs in the intelligent decision-making system.

[0135] The robustness detection device for the intelligent decision-making system provided in this embodiment can execute the robustness detection method for the intelligent decision-making system provided in the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here. Each module in the above-mentioned robustness detection device for the intelligent decision-making system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0136] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above-mentioned method embodiments for detecting robustness of an intelligent decision-making system when executing the computer program.

[0137] For example, Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 6 As shown, the electronic device provided in this embodiment includes: a memory 61 and a processor 62, the memory 61 is used to store computer programs; the processor 62 is used to execute the steps in the robustness detection method of the intelligent decision-making system provided in the above method embodiment when calling the computer program, and its implementation principle and technical effect are similar, which will not be repeated here. Those skilled in the art can understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the robustness detection method of the intelligent decision-making system of any one of the above-mentioned method embodiments are implemented.

[0139] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0140] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0141] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robustness detection method for an intelligent decision-making system, characterized in that: include: Processing the original data in the original data set to generate a detection data set; The detection data set includes: transformed data obtained by processing the original data, adversarial data, and optimized data obtained based on a fuzzy algorithm; Inputting the detection data set into the intelligent decision-making system, and processing the data in the detection data set based on the intelligent decision-making system; Obtaining the neuron coverage, network layer coverage, and neuron pair coverage of the intelligent decision-making system during the processing; The robustness of the intelligent decision-making system is determined based on the neuron coverage, the network layer coverage, and the neuron pair coverage.

2. The method according to claim 1, characterized in that The processing of the original data in the original data set to generate the detection data set includes: Performing linear transformation and / or affine transformation on the original data in the original data set to obtain the transformed data; Inputting the original data into the trained generative adversarial network model for processing, obtaining and outputting the adversarial data; The original data is processed using multi-coverage guided fuzz testing to obtain the optimized data.

3. The method according to claim 1, characterized in that The neuron coverage rate of the intelligent decision-making system in the acquisition process includes: For each neuron in the intelligent decision-making system, determining activation state data of the neuron according to a relationship between an output value of the neuron and an activation threshold; The neuron coverage of the intelligent decision-making system is calculated based on the activation state data of the neurons.

4. The method according to claim 3, characterized in that The calculating the neuron coverage of the intelligent decision-making system based on the activation state data of the neurons includes: Calculating the activated neuron coverage rate, k-node neuron coverage rate, and neuron boundary coverage rate of the intelligent decision-making system according to the activation state data of the neurons; The neuron coverage rate of the intelligent decision-making system is obtained by weighted summing the activated neuron coverage rate, the k-node neuron coverage rate, and the neuron boundary coverage rate.

5. The method according to claim 1, characterized in that: The network layer coverage of the intelligent decision-making system in the acquisition process includes: For each network layer in the intelligent decision-making system, the output value of each neuron in the network layer is obtained, and the proportion of neurons whose output values ​​rank in the top K positions in all neurons in the network layer is determined as the coverage rate of the top K positions of neurons corresponding to the network layer; The network layer coverage rate in the intelligent decision-making system is obtained by weighted summing up the coverage rates of the first K neurons in each network layer.

6. The method according to claim 1, characterized in that The neuron pair coverage of the intelligent decision-making system in the acquisition process includes: For each neuron pair in the intelligent decision-making system, determining the output value, sign, and distance of two neurons in the neuron pair; The neuron pair coverage in the intelligent decision-making system is determined according to the output values, signs, and distances of the two neurons in each neuron pair.

7. The method according to claim 6, characterized in that Determining the neuron pair coverage rate in the intelligent decision-making system according to the output values, signs, and distances of the two neurons in each neuron pair includes: For each neuron pair, if the signs of two neurons in the neuron pair are the same, it is determined that the neuron pair is covered; or, If the signs of the two neurons in the neuron pair are the same and the difference between the output values ​​of the two neurons is greater than a first threshold, it is determined that the neuron pair is covered; or, If the signs of the two neurons in the neuron pair are the same, and the output values ​​of the two neurons are both greater than the second threshold, it is determined that the neuron pair is covered; or, If the difference between the output values ​​of two neurons in the neuron pair is greater than a first threshold, and the output values ​​of the two neurons are both greater than a second threshold, it is determined that the neuron pair is covered; The proportion of covered neuron pairs in the neuron pairs of the intelligent decision-making system is calculated to determine the coverage rate of neuron pairs in the intelligent decision-making system.

8. A robustness detection device for an intelligent decision-making system, characterized in that: include: The raw data processing module is used to process the raw data in the raw data set to generate a detection data set; The detection data set includes: transformed data obtained by processing the original data, adversarial data, and optimized data obtained based on a fuzzy algorithm; A detection data processing module, used for inputting the detection data set into the intelligent decision-making system, and processing the data in the detection data set based on the intelligent decision-making system; A coverage calculation module, used to obtain the neuron coverage, network layer coverage, and neuron pair coverage of the intelligent decision-making system during the processing; A robustness evaluation module is used to determine the robustness of the intelligent decision-making system based on the neuron coverage, the network layer coverage, and the neuron pair coverage.

9. An electronic device, comprising: A memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the robustness detection method for the intelligent decision-making system according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robustness detection method of the intelligent decision-making system described in any one of claims 1 to 7 is implemented.

Images