A coverage analysis method, system, device and storage medium based on propagation-sensitive neurons

Abstract

The application discloses a coverage analysis method and system based on propagation sensitive neurons, a device and a storage medium, and belongs to the technical field of intelligent software coverage testing. The method comprises the following steps: constructing a linear upper and lower bound function of an intermediate layer neuron activation value relative to an input sample, and a linear upper and lower bound function of a neural network output relative to the intermediate layer neuron activation value; respectively calculating the maximum value and the minimum value of the upper and lower bound functions, then taking the difference between the two, to obtain an output interval width function, and respectively calculating the derivative of the perturbation range; performing normalization processing on the gradient norm, and calculating the geometric mean value; constructing a propagation sensitive neuron set of each layer, and performing clustering analysis on the set; using a test data set, evaluating the coverage of the test sample on the propagation sensitive neurons, and obtaining a coverage rate index; and the test coverage index can reveal the coverage degree of the test set on the potential vulnerable area of the model, and significantly improves the reliability and fine-grained ability of model verification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent software coverage testing, and particularly relates to a coverage rate analysis method, system, device and storage medium based on propagation sensitive neurons. BACKGROUND

[0002] Deep neural networks are vulnerable to adversarial perturbations, for example, adding slight perturbations can cause classification models to misclassify. In addition, data set shift can also cause the prediction results of the model to be wrong, which leads to their inability to be deployed in complex and dynamic real-world environments. Therefore, neural network test sufficiency has become a security task that needs to be solved at present, which has challenged the evaluation of the reliability of intelligent systems.

[0003] Unlike traditional code-based software, neural network-based software is a black box and is not interpretable, which also means that traditional software testing methods, such as code coverage, cannot be directly applied to deep learning software. The prior art discloses a test method for deep learning software based on traditional software white box testing methods. The main use is the structure coverage standard, which measures the part of the structure entity (neuron activation, neuron output range, neuron set, etc.) executed by the test suite.

[0004] Deep learning software testing aims to cover more of the decision logic of the neural network, which can detect more error samples. However, these methods do not consider the amplification of neuron perturbation propagation in the neural network, only consider the influence of perturbation on neuron activation value, and do not comprehensively consider the influence of perturbation on neuron activation value and the influence of neuron activation value change on neural network output. SUMMARY

[0005] In view of the problems mentioned in the background, the application provides a coverage rate analysis method, system, device and storage medium based on propagation sensitive neurons, which can accurately identify key neurons in the perturbation propagation link, measure the activation diversity of test data on the combination of propagation sensitive neurons by constructing the PSC index, and realize quantitative evaluation of test effectiveness from the "neural network internal structure level".

[0006] Technical scheme: In order to solve the above technical problems, the technical scheme adopted by the application is as follows:

[0007] A coverage rate analysis method based on propagation sensitive neurons, comprising the following steps:

[0008] S1: using a linear relaxation method, constructing a linear upper and lower bound function of the activation value of the intermediate layer neuron relative to the input sample, and a linear upper and lower bound function of the neural network output relative to the activation value of the intermediate layer neuron;

[0009] S2: Calculate the maximum and minimum of the upper and lower bound functions in S1 respectively and make a difference to obtain a linear function of the width of the intermediate layer neuron activation value interval with respect to the input perturbation range, and a linear function of the output interval width with respect to the perturbation range of the intermediate layer activation value;

[0010] S3: Derive the linear functions in S2 respectively to obtain the gradient information of each neuron in different propagation paths;

[0011] S4: For each neuron in the network, normalize the gradient norm, and then calculate the geometric mean value as the score index of the amplification of the perturbation ability of the neuron in the propagation process;

[0012] S5: According to the score index of S4, select the top k neurons in each layer to form the sensitive propagation neurons, then combine the propagation sensitive neurons of each training set picture to obtain the propagation sensitive neuron set of each layer, and use K-means clustering algorithm to perform clustering analysis on the set;

[0013] S6: Input the test data set, evaluate the coverage of the test sample to the propagation sensitive neurons according to the activation value matching strategy, and then obtain the coverage index of the model.

[0014] As preferred, in S1, the specific implementation process is:

[0015] S11: input the deep neural network model to be tested, the training data set, and set the perturbation range;

[0016] S12: For each input image sample x in the training data set, construct a perturbation set within the set perturbation range;

[0017] S13: Based on the CROWN algorithm, linear upper and lower bound approximation is performed on the activation function of each neuron in each layer of the deep learning model to be tested.

[0018] As preferred, in S2, the specific implementation process is:

[0019] S21: For any layer l of the neural network, extract the linear upper and lower bound functions of the intermediate layer neuron activation value with respect to the input x in S1 and , calculate the maximum and minimum in the input field , obtain the global upper and lower bounds of the intermediate layer neuron activation value with respect to the input x and ; then make a difference between the two to obtain the corresponding activation value interval;

[0020] S22: Based on the linear upper and lower bound functions of the network output with respect to the intermediate layer neuron activation value in S1 With , set the perturbation range of the intermediate layer neuron activation value as , calculate the maximum and minimum values in the perturbation range, and obtain the global upper and lower bounds of the neural network output on the intermediate layer neuron activation value and ; the difference between the two is obtained. The neural network output interval.

[0021] As a preferred, in S3, the specific implementation process is:

[0022] S31: Gradient calculation is performed on the interval width function of the intermediate layer neuron activation value constructed in S2 ;

[0023] The gradient calculation formula is:

[0024] ,

[0025] wherein, represents the activation value output range of the i-th neuron of the k-th layer, represents the perturbation range, represents the linear upper bound coefficient of the input x to the i-th neuron of the k-th layer, represents the linear lower bound coefficient of the input x to the i-th neuron of the k-th layer; q represents the dual norm of the norm of the perturbation range;

[0026] S32: Gradient calculation is performed on the neural network output interval width function constructed in S2 , denoted as:

[0027] ,

[0028] wherein, represents the output range of the j-th component of the model output to the activation value of the i-th neuron of the k-th layer; represents the linear upper bound coefficient of the j-th component of the model output to the activation value of the i-th neuron of the k-th layer; represents the linear lower bound coefficient of the j-th component of the model output to the activation value of the i-th neuron of the k-th layer.

[0029] As a preferred, in S4, the specific implementation process is:

[0030] S41: For each neuron, respectively obtain its gradient information in the two key perturbation propagation paths;

[0031] S42: Standard score normalization is performed on the two gradient norms in S41;

[0032] S43: Geometric mean operation is performed on the two normalized indicators to obtain the final sensitivity score of the neuron.

[0033] As preferred, in S5, the specific implementation process is as follows:

[0034] S51: Sort the neurons of each layer according to the score indicators in S4, select a number of neurons with high ranking as the propagation sensitive neurons, and form the key node set of the layer;

[0035] S52: Repeat S51 for each training image, combine the propagation sensitive neurons of each training set picture, and the sample number of the propagation sensitive neurons exceeding the set threshold value constitutes the propagation sensitive neuron set of each layer;

[0036] S53: Obtain the activation value of each neuron under different input samples, obtain the corresponding cluster center by using clustering algorithm, construct the propagation sensitive neuron cluster center set, and count the total number of hit cluster combinations of all test samples.

[0037] As preferred, in S6, the specific implementation process is as follows:

[0038] S61: Input the test data set to be evaluated and the propagation sensitive neuron cluster center set constructed in S5;

[0039] S62: For each cluster combination in the propagation sensitive neuron cluster center set, based on the distance between the activation value of the test sample on the propagation sensitive neuron and the cluster center, judge whether the test sample covers the combination;

[0040] S63: Count the number of combinations covered by the test set, and then divide by the total number of combinations in the propagation sensitive neuron cluster center set to obtain the propagation sensitive combination coverage rate indicator PSC, which is calculated as follows:

[0041]

[0042] Wherein, PSNCC(j) represents the jth combination in the propagation sensitive neuron cluster center set; y represents the test sample; Y represents the test data set; represents the activation value of input y on the nth neuron; represents the cluster center corresponding to the nth neuron. represents the activation value of input y, the nearest cluster center of the nth neuron.

[0043] ​A coverage analysis system based on propagation sensitive neurons, implementing the coverage analysis method based on propagation sensitive neurons of any of the above, the system comprising a linear upper and lower bound function construction module, an interval width function calculation module, a gradient information acquisition module, a neuron sensitivity scoring module, a propagation sensitive neuron set and clustering module, a coverage evaluation module;

[0044] The linear upper and lower bound function construction module: input the to-be-tested deep neural network model, the training data set and set the perturbation range, construct a perturbation set in the perturbation range for each image sample in the training set, and then based on the CROWN algorithm, linearly relax the activation function of each neuron of each layer of the to-be-tested model to construct linear upper and lower bound functions respectively.

[0045] The interval width function calculation module: based on the upper and lower bound functions, the maximum and minimum values of the corresponding perturbation range are calculated respectively, and two types of linear functions are obtained by difference operation to quantify the correlation between perturbation and interval width.

[0046] The gradient information acquisition module: the two types of linear functions obtained by the interval width function calculation module are differentiated respectively to obtain the gradient information of each neuron in the two key propagation paths.

[0047] The neuron sensitivity scoring module: for each neuron, first extract the gradient information of the two paths obtained by the gradient information acquisition module, and then perform standard score normalization on the two types of gradient norms to eliminate scale differences. Then, the normalized gradient indicators are geometrically averaged to obtain the sensitivity score of the neuron in the propagation process to amplify the perturbation capability.

[0048] The propagation sensitive neuron set and clustering module: according to the sensitivity score of the neuron sensitivity scoring module, the neurons whose sample number exceeds the set threshold are selected to form a propagation sensitive neuron set for each layer. Further, the activation values of each neuron in the set under different input samples are collected, and the K-means clustering algorithm is used to obtain the clustering centers to construct a propagation sensitive neuron clustering center set.

[0049] The coverage evaluation module: input the test data set to be evaluated and the clustering center set constructed by the propagation sensitive neuron set and clustering module, for each clustering combination, judge whether the test sample covers the combination; count the number of covered combinations, calculate the propagation sensitive combination coverage rate index, and realize the quantitative evaluation of the effectiveness of the test set.

[0050] An electronic device, comprising a processor, a memory, a user interface and a network interface, the memory being used to store instructions, the user interface and the network interface being used to communicate with other devices, and the processor being used to execute the instructions stored in the memory to make the electronic device execute the method of any of the above.

[0051] A computer-readable storage medium storing a computer program loadable and executable by a processor to perform the method of any of the above.

[0052] Advantages: Compared with the prior art, the present application has the following advantages:

[0053] (1) The present application can accurately identify the key neurons in the disturbance propagation link: by constructing the linear function of the intermediate neuron activation value interval width to the input, the linear function of the output interval width to the neuron activation value, and combining the gradient norm to construct the disturbance sensitivity score, the propagation-sensitive neurons (Propagation-Sensitive Neurons, PSNs) that play an amplification role in the input disturbance transmission process can be effectively identified. Compared with the existing method of selecting neurons based on heuristic or opaque criteria, the present application provides a clear and computable quantitative standard, and the identification result is more interpretable.

[0054] (2) The present application proposes a test coverage index based on propagation-sensitive combination: by constructing the PSC (Propagation-Sensitive Coverage) index, the activation diversity of the test data on the propagation-sensitive neuron combination is measured, and the quantitative evaluation of the test effectiveness from the "neural network internal structure level" is realized. Compared with traditional precision or input distribution coverage index, this index can better reveal the coverage degree of the test set on the potential vulnerable area of the model, and significantly improve the reliability and fine-grained ability of model verification.

[0055] (3) The present application has strong adaptability and is suitable for various neural network structures; the linear approximation propagation strategy (such as the upper and lower bound estimation based on CROWN) and the gradient norm scoring mechanism in the present application are general computing frameworks, which can be adapted to various mainstream neural network structures such as multilayer perceptron, convolutional neural network, residual network, etc., and have good universality and scalability. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a flowchart of the coverage analysis method based on propagation-sensitive neurons of the present application;

[0057] Figure 2 is a computing framework diagram of the propagation-sensitive neurons of the present application. DETAILED DESCRIPTION

[0058] The present application will be further illustrated below in conjunction with specific embodiments, which are implemented on the premise of the technical scheme of the present application, and it should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0059] As Figure 1As shown, the coverage analysis method based on propagation sensitive neurons provided by the embodiment mainly includes the following steps: first, a linear relaxation method is used to construct a linear upper and lower bound function of the activation value of the intermediate layer neurons with respect to the input sample, and a linear upper and lower bound function of the network output with respect to the activation value of the intermediate layer neurons. Then, the maximum and minimum values of the above upper and lower bound functions are calculated and the difference is obtained, to obtain a linear relationship function between the intermediate layer activation value interval width and the input disturbance range, and a linear function between the output interval width and the intermediate layer disturbance. Further, the derivatives of the two types of functions are calculated to obtain the gradient information of each neuron in different propagation paths. Subsequently, the gradient norm of each neuron in the input propagation path and the output propagation path is calculated respectively, and normalized, and finally the geometric mean of the two is taken as the sensitivity score index of the propagation disturbance ability of the neuron. According to the above score results, the top k neurons in the sensitivity score are selected in each intermediate layer to form propagation sensitive neurons, and then the propagation sensitive neurons of each training set picture are combined to obtain the propagation sensitive neuron set of each layer. And its representative activation center is established by clustering; finally, combined with the test sample, the coverage of the response mode of these propagation sensitive neurons is evaluated, and the disturbance path diversity of the test set is quantified. Specifically, the following steps are included:

[0060] S1: a linear relaxation method is used to construct a linear upper and lower bound function of the activation value of the intermediate layer neurons with respect to the input sample, and a linear upper and lower bound function of the network output with respect to the activation value of the intermediate layer neurons; the specific way is as follows:

[0061] S11: input the deep neural network model to be tested, the training data set, and set the disturbance range;

[0062] The model includes a plurality of levels of neuron nodes and their connection weights; the training data set includes a plurality of input image samples; further, the disturbance range is set , which is used to represent the acceptable disturbance intensity of the input sample.

[0063] S12: for each input image sample x in the training data set, a disturbance set is constructed within the set disturbance range

[0064] The disturbance set can be represented as a closed ball with as the center and as the radius.

[0065] S13: using a CROWN (convex relaxation-based robustness verification method) algorithm to perform linear upper and lower bound approximation on the activation function of each neuron in each layer of the to-be-tested deep learning model;

[0066] Specifically, for any perturbation input x in the perturbation set B, the following linear upper and lower bound functions are constructed, specifically:

[0067] The linear upper bound function is:

[0068]

[0069] The linear lower bound function is:

[0070]

[0071] wherein, and denote the linear upper bound coefficient and the lower bound coefficient of the input x to the i-th neuron in the k-th layer, and both denote the offset term caused by the activation linear relaxation; b (r) denotes the bias term of the r-th layer of the neural network.

[0072] Similarly, given the activation of the i-th neuron in the k-th intermediate layer , the final output of the model can be found. The linear upper and lower bound functions of are as follows:

[0073] The final output of the model The linear upper bound function of is as follows:

[0074]

[0075] The final output of the model The linear lower bound function of is as follows:

[0076]

[0077] wherein, denotes the upper bound linear coefficient of the j-th neuron in the final output to the r-th activation in the k-th intermediate layer from the (k+1)-th layer, denotes the lower bound linear coefficient of the j-th neuron in the final output to the r-th activation in the k-th intermediate layer from the (k+1)-th layer, and m denotes the total number of layers of the neural network; and respectively denote the bias correction terms introduced by the upper and lower bound linearization, b (r) denotes the bias term of the r-th layer of the neural network.

[0078] which can be simplified as:

[0079]

[0080]

[0081] where, denotes the linear upper bound coefficient of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; denotes the linear lower bound coefficient of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; denotes the bias term of the linear upper bound of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer, denotes the bias term of the linear lower bound of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; denotes the activation value of the i-th neuron in the k-th intermediate layer.

[0082] S2: Calculate the maximum and minimum of the upper and lower bound functions and subtract them to obtain the linear function of the interval width of the intermediate layer neuron activation value with respect to the input perturbation range, and the linear function of the output interval width with respect to the perturbation range of the intermediate layer activation value;

[0083] By respectively taking the maximum and minimum of the above upper and lower bound functions, and then subtracting them, the linear function of the interval width of the intermediate layer neuron activation value with respect to the input perturbation range, and the linear function of the output interval width with respect to the perturbation range of the intermediate layer activation value are obtained; the derivatives of the two with respect to the input perturbation range and the activation value perturbation range are obtained, and the gradient information of each neuron in the two propagation paths is obtained; the specific method is as follows:

[0084] S21: For any layer l of the neural network, extract the linear upper and lower bound functions of the intermediate layer neuron activation value with respect to the input x obtained in S1 and , calculate the maximum and minimum in the input domain to obtain the global upper and lower bounds of the intermediate layer neuron activation value with respect to the input x and ; then subtract them to obtain the corresponding activation value interval;

[0085] Specifically: use the norm maximization conclusion in convex optimization, that is, for a linear function, , constrained on , its maximum is:

[0086]

[0087] Similarly, the minimum value is:

[0088]

[0089] where, denotes the perturbation range, p denotes the norm of the perturbation range, and q denotes the dual norm of p; denotes the coefficient matrix of the linear function; x denotes the image with the perturbation added to x0; denotes the activation value of the neuron after perturbation; x0denotes the original image.

[0090] Therefore, the maximum value of the upper bound is:

[0091] ,

[0092] The minimum value of the lower bound is:

[0093] ,

[0094] where, denotes the linear upper bound coefficient of input x to the i-th neuron in the k-th layer, denotes the input perturbation range, q denotes the dual norm of p, i.e., q denotes the dual norm of the norm of the perturbation range; x0denotes the original image; denotes the offset term caused by the activation linear relaxation; b (r) denotes the bias term of the r-th layer of the neural network; denotes the linear lower bound coefficient of input x to the i-th neuron in the k-th layer, denotes the offset term caused by the activation linear relaxation.

[0095] The difference between the two gives the corresponding activation value interval:

[0096]

[0097] Specifically:

[0098]

[0099] where, and denote the linear upper bound coefficient and the linear lower bound coefficient of input x to the i-th neuron in the k-th layer, and denote the offset term caused by the activation linear relaxation; b (r) denotes the bias term of the r-th layer of the neural network; and denote the linear upper and lower bound functions of the intermediate layer neuron activation value in S1 with respect to input x. denotes the output range of the k-th layer i-th neuron activation value; denotes the perturbation range.

[0100] S22: Similarly, based on the linear upper and lower bound functions of the network output with respect to the intermediate layer neuron activation values calculated in S1 and , let the perturbation range of , define the perturbation set as:

[0101] ,

[0102] wherein, denotes the original activation value of the neuron, denotes the activation value of the neuron after perturbation; denotes the set of real numbers; denotes the perturbation range of the intermediate layer neuron activation value, i.e. the perturbation range of denotes the activation value of the i-th neuron in the k-th layer of the intermediate layer.

[0103] Using the previous conclusions, the closed-form expressions of the global linear upper and lower bounds of the j-th component of the model output when only perturbing the activation value of the i-th neuron in the k-th layer are obtained, which are respectively:

[0104] ,

[0105] ,

[0106] Subtracting the two expressions gives the maximum interval of the output, which is specifically:

[0107] ,

[0108] wherein, denotes the output range of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; denotes the linear upper bound coefficient of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; denotes the linear lower bound coefficient of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; denotes the original activation value of the neuron (the activation value of the i-th neuron in the k-th layer); the offset term in the linear upper bound of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; denotes the offset term in the linear upper bound of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer. denotes the perturbation range of the intermediate layer neuron activation value, i.e. the perturbation range of and denote the closed-form expressions of the global linear upper and lower bounds of the j-th component of the model output when only the i-th neuron activation value of the k-th layer is perturbed, respectively.

[0109] The obtained and will serve as the basis for subsequent sensitivity score calculations, respectively characterizing the perturbation amplification ability from the input layer to the intermediate layer and from the intermediate layer to the output layer.

[0110] S3: Derive the function in S2 to obtain gradient information in the gradient of each neuron in different propagation paths;

[0111] S31: Perform gradient calculation on the interval width function of the activation value of the intermediate layer neuron constructed in S2 to obtain the derivative of the perturbation radius , which is specifically:

[0112] The interval width function is:

[0113] ;

[0114] The gradient calculation formula is:

[0115] ,

[0116] wherein, denotes the output range of the activation value of the i-th neuron of the k-th layer, denotes the perturbation range, denotes the linear upper bound coefficient of the input x to the i-th neuron of the k-th layer, denotes the linear lower bound coefficient of the input x to the i-th neuron of the k-th layer; q denotes the dual norm of the perturbation range norm, i.e., q denotes the dual norm of p.

[0117] It represents the sensitivity of the neuron output range to the perturbation range , which reflects the amplification ability of the neuron to the input perturbation. The above gradient vector characterizes the intensity of the change in the activation value interval of the neuron caused by the input perturbation during the propagation to the l-th layer, reflecting the contribution degree of the input perturbation range to the output change amplitude of the layer.

[0118] S32: Perform gradient calculation on the interval width function of the neural network output constructed in S2 to obtain the gradient of the perturbation radius , which is:

[0119]

[0120] wherein, denotes the linear output range of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; denotes the linear upper bound coefficient of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; denotes the linear lower bound coefficient of the j-th component of the model output with respect to the activation value of the i-th neuron in the k-th layer; is the perturbation range of the intermediate layer activation value.

[0121] It measures the sensitivity of the j-th output of the neural network to the i-th neuron of the intermediate layer. The gradient vector reflects the direct amplification ability of the perturbation size of the i-th neuron in the k-th layer to the output interval, and measures the influence degree of the perturbation of the neuron in the layer to the stability of the output.

[0122] The above two gradient results are the core inputs for constructing the importance score function in the subsequent steps, which are used to comprehensively model the key role of each intermediate layer neuron in the input perturbation-intermediate perturbation-output change conduction path.

[0123] S4: For each neuron in the network, the above two gradients are normalized, and then the geometric mean value is calculated as the score index of the amplification of the perturbation ability in the propagation process; the specific way is as follows:

[0124] S41: For each neuron, the gradient information in the two key perturbation propagation paths is obtained respectively;

[0125] Path one: the gradient corresponding to the propagation of the input perturbation to the neuron (input-activation value), denoted as .

[0126] Path two: the gradient corresponding to the propagation of the activation value perturbation of the neuron to the network output (activation value-output), denoted as .

[0127] S42: In order to eliminate the scale difference between different neurons, the and are respectively normalized by standard score (z-score normalization) to obtain z1 and z2, and the calculation formula is as follows:

[0128] ,

[0129] ,

[0130] Wherein, μ1 represents the mean value of the gradient set of path one, μ2 represents the mean value of the gradient set of path two, and σ1, σ2 represent the standard deviation.

[0131] S43: Then, the geometric mean operation is performed on the two normalized indicators z1 and z2 to obtain the final sensitivity score score of the neuron, and the specific calculation formula is:

[0132]

[0133] wherein, is a numerical stability constant, which is used to avoid the occurrence of non-number (NaN) in the calculation of decimal square root. The final score represents the comprehensive ability of the neuron to amplify the disturbance in both input propagation and output feedback directions, which can be used for subsequent path construction and screening.

[0134] S5: According to the score index of S4, the top-k neurons in each layer are selected to form the propagation-sensitive neurons, and then the propagation-sensitive neurons of each training set picture are combined to obtain the propagation-sensitive neuron set of each layer, and the representative activation center is established by clustering.

[0135] According to the score index of S4, all neurons in each layer are ranked or selected, and the top-k neurons are called propagation-sensitive neurons (Propagation-Sensitive Neurons), which amplify the output interval width.

[0136] The propagation-sensitive neurons of each training set picture are combined to obtain the propagation-sensitive neuron set, and the activation value of each neuron is obtained. The activation value of each neuron is clustered by K-means clustering method to form K clusters. PSNCC is a set formed by combining the cluster centers of all propagation-sensitive neurons; the specific method is as follows:

[0137] S51: According to the score index obtained by S4, all neurons in each layer are sorted according to Score, and the top-k neurons are selected as the propagation-sensitive neurons (Propagation-Sensitive Neurons) of the layer. This kind of neuron has stronger output interval width amplification ability and is regarded as the key node of disturbance propagation;

[0138] S52: The operation of S51 is repeated for each training image x to obtain the corresponding propagation-sensitive neuron set respectively; finally, the propagation-sensitive neurons corresponding to all training samples are combined, and the propagation-sensitive neurons whose sample number exceeds the set threshold constitute the overall propagation-sensitive neuron set N PSN , which is specifically:

[0139]

[0140] wherein, a subset of propagation-sensitive neurons representing the image x, representing a training dataset.

[0141] S53: After obtaining the set of propagation-sensitive neurons, the activation values of these neurons under different input samples are calculated;

[0142] For each propagation-sensitive neuron j, a set of its activation values under all training samples is collected and a K-means clustering algorithm is used to analyze the clustering of the set, obtaining K cluster centers, specifically:

[0143]

[0144] wherein, Centroids(j) represents the activation value cluster center of propagation-sensitive neuron j.

[0145] Combining the cluster centers of all propagation-sensitive neurons, a set of propagation-sensitive neuron cluster centers (Propagation-Sensitive Neuron Cluster Centers, PSNCC) is formed, and the total number of cluster combinations hit by all test samples is counted, specifically:

[0146]

[0147] wherein, Centroids(j) represents the activation value cluster center of propagation-sensitive neuron j.

[0148] This set of PSNCCs depicts the typical response patterns of the model in the dimension of key neurons, which is used for subsequent sample coverage calculation.

[0149] S6: Input the test dataset, and evaluate the coverage of the test sample to the propagation-sensitive neurons according to the activation value matching strategy, to obtain the coverage rate of the model;

[0150] Input the test dataset, for each neuron in the set of propagation-sensitive neurons, use the cluster center matching method to check whether it is activated by a certain test set picture, and measure how many combinations in the PSNCC are covered by the samples in the input set Y. If the neuron activation value of the input y is closest to the center of a certain PSNCC combination, it is considered to "cover" this combination. Each test input y is mapped to the closest combination in a PSNCC (i.e., which combination it belongs to). Define the propagation-sensitive combination coverage rate indicator PSC as the number of combinations covered by the test sample divided by the total number of combinations in the PSNCC; the specific way is as follows:

[0151] S61: input a test data set to be evaluated and the set of propagation-sensitive neuron cluster centers PSNCC constructed in S5;

[0152] S62: for each cluster combination in PSNCC (denoted as PSNCC(j), representing the jth combination, corresponding to one or more propagation-sensitive neurons and their cluster centers), define the decision rule of whether it is covered by the test data set Y as follows:

[0153] For each test sample y e Y, input it into the neural network to obtain the corresponding propagation-sensitive neuron activation value vector and map it to the combination PSNCC(j) with the smallest distance in all PSNCC, that is:

[0154] If for a combination PSNCC(j), the activation value of each neuron n in the combination is closest to a cluster center , then the test sample y is considered to successfully cover the combination PSNCC(j).

[0155] Each test sample at most contributes once to the coverage of one combination PSNCC(j) that has not been covered. If the combination has been covered by other test samples before, the new coverage is not counted this time.

[0156] The propagation-sensitive coverage indicator PSC is calculated as follows:

[0157]

[0158] Where PSNCC(j) represents the jth combination in the set of propagation-sensitive neuron cluster centers; y represents a test sample; Y represents a test data set; represents the activation value of input y on the nth neuron; represents the cluster center corresponding to the nth neuron; represents the closest cluster center of the activation value of the nth neuron for input y.

[0159] The effectiveness and efficiency of the method of the present application are verified by the following experiments:

[0160] The evaluation indicators are the coverage of test samples and the coverage after adding adversarial samples.

[0161] The present application selects the MNIST data set and LeNet-5 model.

[0162] MNIST Dataset: The MNIST (Modified National Institute of Standards and Technology) dataset is a classic image classification dataset for handwritten digit recognition, containing 70,000 28x28 pixel grayscale images, divided into 60,000 training images and 10,000 test images. Each image represents a digit (0 to 9) and is widely used for training and evaluating machine learning and deep learning models.

[0163] LeNet-5 Model: LeNet-5 is a classic convolutional neural network model originally designed for handwritten digit recognition tasks, particularly on the MNIST dataset.

[0164] Comparison methods: NC, SNAC, NBC, KMNC, TKNC, TKNP, IDC.

[0165] NC: Neuron Coverage (NC) is a classic performance indicator in neural network testing that integrates coverage ideas, commonly used to measure whether test cases can correctly explore the internal state of the model. Neuron coverage essentially refers to the ratio of the number of activated neurons in a set of test data during testing to the total number of neurons in the network. The higher the value, the more comprehensive the test.

[0166] SNAC: Strong Neuron Activation Coverage (SNAC), for neurons whose output exceeds the maximum value, they are considered as overactive neurons, which pass useful decision information within the model, called "strong neurons". Measures how many boundary cases are covered by a given test input t.

[0167] NBC: Neuron Boundary Coverage (NBC) is defined as the ratio of covered boundary cases to the total number of boundary cases, used to measure how many boundary case regions (upper and lower boundary values) are covered by a given test input set t.

[0168] KMNC: K-Multisection Neuron Coverage (KMNC),

[0169] The output of the neuron is further refined from the original "activated" and "non-activated" two simple states to the value interval defined by the [minimum value, maximum value] output by the training stage, and the interval is divided into several small sections. Then input the test data, count which small sections the output of each neuron falls into, and finally calculate the proportion of covered small sections to the total number of sections.

[0170] TKNC: (top-k neuron coverage, or TKNC) refers to how many neurons on each layer have ever been the most active, defined as the ratio of the total number of top-k neurons on each layer to the total number of neurons in the DNN.

[0171] TKNP: (top-k neuron group, or TKNP), given a test input, the sequence of top-k neurons on each layer also forms a pattern, i.e., the top-k neuron pattern represents the activation of different kinds of neurons from the top super-active neurons on each layer.

[0172] IDC: Given a set of inputs, it can measure how much it covers the important neuron clusters, called importance-driven coverage (IDC), the higher the IDC, the more sufficient the combination of activation value clusters that trigger those neurons.

[0173] The coverage of the invention based on the propagation-sensitive neuron is referred to as PSC. PGD, FGSM and BIM are all existing adversarial samples.

[0174] Table 1 Coverage of the invention on the original test set and after adding adversarial samples (%)

[0175]

[0176] The results of Table 1 show that the coverage of the method of the invention after adding adversarial samples is better than the six baseline methods, indicating that the selected neurons of the method are more sensitive to adversarial samples.

[0177] The results show that different coverage analysis methods perform differently after adding adversarial samples. In general, it can be found that the trend of PSC is as expected, i.e., the coverage can be increased after adding adversarial samples. The coverage of NC, KMNC, NBC and SNAC is not significantly increased after adding adversarial samples, indicating that they lack relevance to adversarial samples. The coverage of TKNC and IDC is greatly improved after adding adversarial samples, indicating that more neurons are activated by adversarial samples, but it is still lower than PSC proposed by the invention, so the selected neurons of the invention are more easily activated by adversarial samples, which can be used to test more defect areas of neural networks and improve the sufficiency of neural network testing.

[0178] Overall, the present application proposes a coverage analysis method based on propagation sensitive neurons. First, the linear relaxation method is used to construct the linear upper and lower bound functions of the activation value of the intermediate layer neurons with respect to the input sample, and the linear upper and lower bound functions of the network output with respect to the activation value of the intermediate layer neurons. Then, the maximum and minimum values of the above upper and lower bound functions are calculated and the difference is obtained, and the linear relationship function between the interval width of the intermediate layer activation value and the input perturbation range, and the linear function between the output interval width and the intermediate layer perturbation are obtained. Further, the gradient information of each neuron in different propagation paths is obtained by taking the derivative of the two types of functions. Then, the gradient norm of each neuron in the input propagation path and the output propagation path is calculated respectively, and normalized, and finally the geometric mean of the two is taken as the sensitivity score index of the propagation disturbance ability of the neuron.

[0179] According to the above scoring results, the top k neurons in the sensitivity score in each intermediate layer are selected to form propagation sensitive neurons, and then the propagation sensitive neurons of each training set picture are combined to obtain the propagation sensitive neuron set of each layer. And its representative activation center is established by clustering; finally, combined with the test sample, the coverage of the response mode of these propagation sensitive neurons is evaluated, and the diversity of the perturbation path of the test set is quantified. The present application is more sensitive to adversarial samples than other coverage analysis methods based on neuron structure.

[0180] Based on the above method, the present application provides a coverage analysis system based on propagation sensitive neurons, which comprises a linear upper and lower bound function construction module, an interval width function calculation module, a gradient information acquisition module, a neuron sensitivity scoring module, a propagation sensitive neuron set and clustering module, and a coverage evaluation module.

[0181] The linear upper and lower bound function construction module: input the deep neural network model to be tested, the training data set and set the perturbation range, construct the perturbation set within the perturbation range for each image sample in the training set, and then based on the CROWN algorithm, linearly relax the activation function of each neuron in each layer of the model to be tested, and construct the linear upper and lower bound functions of the "activation value of the intermediate layer neuron with respect to the input sample" and the "output of the neural network with respect to the activation value of the intermediate layer neuron", respectively, to lay the foundation for subsequent interval calculation.

[0182] The interval width function calculation module: for the two types of upper and lower bound functions constructed by the linear upper and lower bound function construction module, the maximum and minimum values within the corresponding perturbation range are calculated respectively, and the difference is obtained to obtain two types of linear functions: "linear function of the interval width of the activation value of the intermediate layer neuron with respect to the input perturbation range" and "linear function of the output interval width with respect to the perturbation range of the intermediate layer activation value", which quantifies the correlation between perturbation and interval width.

[0183] Gradient information acquisition module: the two types of linear functions obtained by the interval width function calculation module are differentiated respectively to obtain the gradient information of each neuron in the two key propagation paths: one is the gradient of the intermediate layer neuron activation value interval width function to the input disturbance range (reflecting the influence of input disturbance on neuron activation value), and the other is the gradient of the neural network output interval width function to the intermediate layer activation value disturbance range (reflecting the influence of neuron activation value disturbance on output).

[0184] Neuron sensitivity scoring module: for each neuron, the two path gradient information obtained by the gradient information acquisition module is extracted, and the two types of gradient norms are normalized by standard scores to eliminate scale differences, and then the normalized gradient indicators are geometrically averaged to obtain the sensitivity score of the neuron in the propagation process to amplify the disturbance ability, which is used as the basis for screening key neurons.

[0185] Propagation sensitive neuron set and clustering module: according to the sensitivity score of the neuron sensitivity scoring module, the neurons in each layer are sorted and the top k neurons are selected as propagation sensitive neurons; the propagation sensitive neurons corresponding to all training set pictures are merged, and the neurons with a sample quantity exceeding a set threshold are selected to form a propagation sensitive neuron set of each layer; further, the activation values of each neuron in the set under different input samples are collected, and the K-means clustering algorithm is used to obtain the clustering centers to construct a propagation sensitive neuron clustering center set.

[0186] Coverage evaluation module: input the test data set to be evaluated and the clustering center set constructed by the propagation sensitive neuron set and clustering module, for each clustering combination, based on the distance between the activation value of the test sample on the propagation sensitive neuron and the clustering center, judge whether the test sample covers the combination; the number of covered combinations is counted, and the propagation sensitive combination coverage index (PSC) is calculated by "covered combination number / total combination number of clustering center set", to realize the quantitative evaluation of the effectiveness of the test set.

[0187] The method and system of the application are applied to the field of intelligent software coverage testing, and the test coverage index based on the propagation sensitive combination proposed by the application can measure the activation diversity of the test data on the propagation sensitive neuron combination, and realize the quantitative evaluation of the effectiveness of the test from the "neural network internal structure level". Compared with the traditional precision or input distribution coverage index, the index can better reveal the coverage degree of the test set on the potential vulnerable area of the model, and significantly improve the reliability and fine-grained ability of model verification.

[0188] The embodiment of the application provides an electronic device, which comprises at least one processor, at least one network interface, a user interface, a memory and at least one communication bus.

[0189] A communication bus is used to realize the connection communication between the components.

[0190] The user interface can include a display screen, a camera, and the optional user interface can further include a standard wired interface, a wireless interface.

[0191] The network interface can optionally include a standard wired interface, a wireless interface.

[0192] The processor can include one or more processing cores. The processor connects various parts within the entire server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of the hardware forms of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), Programmable Logic Array (PLA). The processor can integrate one or a combination of central processing units (CPUs), graphics processing units (GPUs), and modems. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing the content required to be displayed on the display screen; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but be realized by a separate chip.

[0193] The memory can include a random access memory (RAM) and a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets, or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, etc.; and the data storage area can store data involved in the various method embodiments described above, etc. The memory can also optionally be at least one storage device located away from the aforementioned processor. The memory as a computer storage medium can include an operating system, a network communication module, a user interface module, and an application program of the coverage analysis method based on the propagating sensitive neuron.

[0194] In the electronic device of the embodiment, the user interface is mainly used to provide an interface for the user to input, and obtain data input by the user; and the processor can be used to call an application program of the coverage analysis method based on the propagating sensitive neuron stored in the memory, and when executed by one or more processors, make the electronic device execute the method described in one or more of the above embodiments.

[0195] An electronic device readable storage medium stores instructions. When executed by one or more processors, make the electronic device execute the method described in one or more of the above embodiments.

[0196] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for coverage analysis based on propagating sensitive neurons, characterized in that: The method comprises the following steps: S1: using a linear relaxation method, constructing a linear upper and lower bound function of the activation value of the intermediate layer neuron with respect to the input sample, and a linear upper and lower bound function of the neural network output with respect to the activation value of the intermediate layer neuron; S2: calculating the maximum and minimum values of the upper and lower bound functions in S1 respectively and taking the difference to obtain a linear function of the interval width of the activation value of the intermediate layer neuron with respect to the input disturbance range, and a linear function of the output interval width with respect to the disturbance range of the activation value of the intermediate layer; S3: respectively taking the derivative of the linear functions in S2 to obtain gradient information of each neuron in different propagation paths; S4: normalizing the gradient norm in step S3, and then calculating the geometric mean value as a scoring index of the amplification of the disturbance ability of the neuron in the propagation process; S5: according to the scoring index of S4, selecting the top k neurons in each layer to form a sensitive propagation neuron, then merging the propagation sensitive neurons of each training set picture to obtain a propagation sensitive neuron set of each layer, and performing clustering analysis on the set by using a K-means clustering algorithm; The sub-steps of S5 are: S51: sorting the neurons in each layer according to the score of S4, and selecting a certain number of neurons with high ranking as propagation sensitive neurons to form a key node set of the layer; S52: repeating the operation of S51 for each training image, merging the propagation sensitive neurons of each training set picture, and the propagation sensitive neuron set of each layer is formed by the propagation sensitive neurons whose sample number exceeds a certain threshold; S53: obtaining the activation value of each neuron under different input samples, obtaining the corresponding clustering center by using a clustering algorithm, constructing a propagation sensitive neuron clustering center set, and counting the total number of hit clustering combinations of all test samples; S6: inputting a test data set, evaluating the coverage of the test sample on the propagation sensitive neuron according to the activation value matching strategy, and then obtaining the coverage rate index of the model; The sub-steps of S6 are: S61: inputting the test data set to be evaluated and the propagation sensitive neuron clustering center set constructed in S5; S62: for each clustering combination in the propagation sensitive neuron clustering center set, judging whether the test sample covers the combination based on the distance between the activation value of the test sample on the propagation sensitive neuron and the clustering center; S63: counting the number of combinations covered by the test set, and then dividing by the total number of combinations in the propagation sensitive neuron clustering center set to obtain the propagation sensitive combination coverage rate index PSC, which is calculated as follows: ， wherein PSNCC(j) represents the jth combination in the set of propagation sensitive neuron cluster centers; y represents a test sample; Y represents a test dataset; represents the activation value of the input y on the nth neuron; represents the cluster center corresponding to the nth neuron; represents the activation value of the input y, the most recent cluster center of the nth neuron.

2. The method of claim 1, wherein: In S1, the specific implementation process is: S11: inputting a deep neural network model to be tested, a training data set, and setting a disturbance range; S12: For each input image sample x in the training data set, a set of perturbations is constructed within a set perturbation range ; S13: based on the CROWN algorithm, approximating the linear upper and lower bounds of the activation function of each neuron in each layer of the deep learning model to be tested.

3. The method of claim 1, wherein: In S2, the specific implementation process is: S21: For any layer l of the neural network, extract the linear upper and lower bound functions of the intermediate layer neuron activation values with respect to the input x in S1 With , compute the maximum and minimum values in the input domain, obtaining the global upper and lower bounds of the intermediate layer neuron activation values with respect to the input x And ; then subtract the two, obtaining the corresponding activation value interval; the perturbation set represents a closed ball centered at with a radius of under a certain norm;​ S22: output linear upper and lower bound functions of the intermediate layer neuron activation values based on the network in S1 With , let the perturbation range of the intermediate layer neuron activation values be , calculate the maximum and minimum values in the perturbation range to obtain global upper and lower bounds of the neural network output on the intermediate layer neuron activation values and ; The difference between the two is obtained.

4. The method of claim 1, wherein: In S3, the specific implementation process is: S31: interval width function on the intermediate layer neuron activation values constructed in S2 performing gradient computation; The gradient calculation formula is: ， wherein, represents the activation value output range of the i-th neuron of the k-th layer, represents the perturbation range, represents the linear upper bound coefficient of the input x to the i-th neuron of the k-th layer, represents the linear lower bound coefficient of the input x to the i-th neuron of the k-th layer; q represents the dual norm of the norm of the perturbation range; S32: constructing a neural network output interval width function on the basis of the neural network constructed in S2 The gradient calculation is denoted as: ， wherein, represents an output range of the model outputting the jthcomponent for the activation value of the ithneuron of the kthlayer; represents a linear upper bound coefficient of the model outputting the jthcomponent for the activation value of the ithneuron of the kthlayer; represents a linear lower bound coefficient of the model outputting the jthcomponent for the activation value of the ithneuron of the kthlayer.

5. The method of claim 1, wherein: In S4, the specific implementation process is: S41: for each neuron, the gradient information in step S3 is obtained respectively; S42: standard score normalization is performed on the two gradient norms in S41; S43: geometric mean operation is performed on the two normalized indicators to obtain the final sensitivity score of the neuron.

6. A coverage analysis system based on propagating sensitive neurons, implementing the method of coverage analysis based on propagating sensitive neurons according to any one of claims 1 to 5, characterized in that: The system comprises a linear upper and lower bound function construction module, an interval width function calculation module, a gradient information acquisition module, a neuron sensitivity scoring module, a propagation sensitive neuron set and clustering module, and a coverage rate evaluation module. The linear upper and lower bound function construction module: input the to-be-tested deep neural network model, the training data set, and set the perturbation range, construct a perturbation set within the perturbation range for each image sample in the training set, and then based on the CROWN algorithm, linearly relax the activation function of each neuron of each layer of the to-be-tested model to construct linear upper and lower bound functions respectively. The interval width function calculation module: based on the upper and lower bound functions, the maximum and minimum values within the corresponding perturbation range are calculated respectively, and the difference operation is performed to obtain two types of linear functions, which quantize the correlation between perturbation and interval width. The gradient information acquisition module: the two types of linear functions obtained by the interval width function calculation module are differentiated to obtain the gradient information of each neuron in the two key propagation paths. The neuron sensitivity scoring module: for each neuron, the gradient information of the two paths obtained by the gradient information acquisition module is extracted, standard score normalization is performed on the two types of gradient norms to eliminate the scale difference, and then geometric mean operation is performed on the normalized gradient indicators to obtain the sensitivity score of the neuron in the propagation process. The propagation sensitive neuron set and clustering module: according to the sensitivity score of the neuron sensitivity scoring module, the neurons whose sample quantity exceeds the set threshold are selected to form the propagation sensitive neuron set of each layer; further, the activation values of each neuron in the set under different input samples are collected, the K-means clustering algorithm is used to obtain the clustering center, and the propagation sensitive neuron clustering center set is constructed. The coverage rate evaluation module: input the test data set to be evaluated and the clustering center set constructed by the propagation sensitive neuron set and clustering module, for each clustering combination, judge whether the test sample covers the combination; count the number of covered combinations, calculate the propagation sensitive combination coverage rate index, and realize the quantitative evaluation of the effectiveness of the test set.

7. An electronic device, comprising: The electronic device comprises a processor, a memory, a user interface, and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer program stored in the memory can be loaded and executed by the processor to perform the method of any one of claims 1-5.

Images