Intelligent software test sufficiency evaluation method

Abstract

The invention discloses an intelligent software test sufficiency evaluation method, which relates to the technical field of software testing, and comprises the following steps: step 1, constructing a test case data set and a training data set; 2, constructing a neural network training environment and a neural network testing environment; 3, training the intelligent software in a neural network training environment by using the training data set to obtain a to-be-tested neural network; 4, performing neuron coverage test, path coverage test and accident coverage test on the to-be-tested neural network by using the test case data set in the neural network test environment to obtain a neuron coverage rate, a neuron path coverage rate and an accident coverage rate respectively; and 5, comparing and judging according to the neuron coverage rate, the neuron path coverage rate, the accident coverage rate and a set coverage rate threshold value to obtain a sufficiency evaluation result. According to the method, effective test sufficiency evaluation is implemented on the intelligent software based on neurons, paths and accidental coverage.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and more specifically to an intelligent software testing adequacy evaluation method. Background Art

[0002] The core component of intelligent software is the deep neural network model. Through the training and reasoning of the deep neural network model, intelligent functions are realized. Therefore, it is crucial to effectively ensure the quality of intelligent software, especially the deep neural network model. Software testing is a key means to discover defects in intelligent software and ensure quality, and it is also an important means to improve the quality of intelligent software. Due to the complexity and uncertainty of the deep neural network model, it is an important means to fully test the general behavior and boundary behavior of the model to ensure the quality of the model. However, the deep neural network model does not follow the programming paradigm of traditional software. Its decision logic is learned from the training set, which is reflected in the weights of the edges connecting neurons. It has highly complex data dependencies and inherent behavioral uncertainty. Any small disturbance in the sample may cause the model to make incomprehensible erroneous behaviors. The decision logic of traditional software is analyzed through control flow graphs, but the decision logic of intelligent software and algorithms is usually reflected in the weights of the edges connecting neurons in different layers and the activation function. The decision logic of the deep neural network model cannot be analyzed through program analysis methods such as control flow and data flow analysis. The model input has the characteristics of high dimension and complex features, which leads to the traditional software test data generation technology guided by maximizing statement or branch coverage is no longer applicable to deep neural network models.

[0003] Test coverage is an important means of measuring test adequacy. Traditional software testing has formed mature test methods and test measurement systems (such as statement coverage, judgment coverage, condition coverage, etc.). Due to the characteristics of intelligent software, traditional software test adequacy evaluation cannot be transplanted to intelligent software. Data-driven characteristics, neuron-based model structure and framework-based program development methods break the basic assumptions in traditional structured coverage standards. From the survey of intelligent software testing technology, the current research on test adequacy of intelligent software is mainly focused on the research of neuron coverage methods, and neuron coverage, symbol coverage, combination testing and other technologies have been proposed. Researchers have also proposed some innovative coverage indicators such as path coverage and contribution coverage. However, the above indicators cannot fully evaluate the test adequacy of intelligent software, and the test adequacy indicator system based on coverage is still missing. Due to the lack of a system, the test sample generation technology based on coverage may cause the generated data to deviate from the original distribution. Intelligent software lacks efficient test adequacy evaluation methods, which leads to insufficient and incomplete intelligent software test verification.

[0004] Therefore, how to generate test data and test coverage for intelligent software and improve the accuracy of intelligent software test adequacy assessment is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides an intelligent software test adequacy evaluation method. For the intelligent software test adequacy evaluation, an intelligent software test adequacy evaluation system is formed based on neuron paths and unexpected coverage to implement effective test adequacy evaluation of intelligent software.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] An intelligent software testing adequacy evaluation method comprises the following steps:

[0008] Step 1: Build test case dataset and training dataset;

[0009] Step 2: Build an intelligent software testing environment, including a neural network training environment and a neural network testing environment;

[0010] Step 3: Use the training data set to train the neural network of the intelligent software in the neural network training environment to obtain network parameters, substitute the network parameters into the neural network to obtain the neural network to be tested; train and debug the neural network of the intelligent software, complete the test of algorithm convergence, efficiency and correctness, and determine the parameters;

[0011] Step 4: In the neural network test environment, use the test case data set to perform neuron coverage test, path coverage test and accidental coverage test on the neural network to be tested, and obtain neuron coverage rate, neuron path coverage rate and accidental coverage rate respectively;

[0012] Step 5: Compare and judge based on neuron coverage, neuron path coverage, unexpected coverage and the set coverage threshold to obtain the adequacy evaluation result.

[0013] Preferably, when performing neuron coverage test in step 4, a structured coverage analysis based on neuron activation value is performed according to the neuron activation coverage criterion and the neuron network area coverage criterion to obtain neuron coverage, where the neuron coverage includes neuron activation coverage and neuron network area coverage; the specific process is as follows:

[0014] Step 411: input the test samples in the test case data set into the neural network to be tested, and obtain the neuron activation value of the neural network to be tested; select test data of several test cases from the test case data set as test samples; the test case also includes expected output and steps;

[0015] Step 412: Calculate the neuron activation value distribution or the change of the neuron activation value of the adjacent layer according to the neuron activation value;

[0016] Step 413: according to the distribution of neuron activation values, changes in the activation values ​​of neurons in adjacent layers and the set activation threshold, count the activated neurons in each layer of the network structure in the neural network, calculate the ratio of the number of activated neurons to the number of all neurons in the neural network to be tested, and obtain the neuron activation coverage;

[0017] Step 414: Input the test samples in the test case data set into the neural network to be tested, obtain the executed neuron working area, calculate the ratio of the executed neuron working area to the total neuron working area in the neural network to be tested, and obtain the neuron network area coverage.

[0018] Preferably, the specific process of performing the path coverage test in step 4 to obtain the neuron path coverage is:

[0019] Step 421: extracting the key decision path of training for each training sample in the training data set of the neural network training process in step 3 respectively;

[0020] Step 422: extracting the key decision path of each test sample in the test case data set after inputting it into the neural network to be tested for execution;

[0021] Step 423: clustering and abstracting the training key decision paths of all training samples into k clusters;

[0022] Step 424: abstract paths for all clusters, and merge the key decision paths in each cluster into an abstract key decision path;

[0023] Step 425: Calculate the similarity between the test key decision path in step 422 and the abstract key decision path of each cluster, and calculate the distance between the test sample and the training sample;

[0024] Step 426: Eliminate duplicate data in all similarities and distances, count the number of similarities and distances, obtain the path coverage value, calculate the ratio of the path coverage value to the number of key decision paths tested, and obtain the neuron path coverage rate.

[0025] Preferably, the specific process of performing the accidental coverage test in step 4 to obtain the accidental coverage rate is:

[0026] Step 431: A kernel density estimation method is used to calculate the likelihood-based surprise sufficiency of the test samples in the test case data set, and the expression is:

[0027]

[0028] Among them, x represents the test sample, represents the neuron output value vector composed of the neuron output values ​​corresponding to the important neuron set of the target layer in the neural network to be tested; T represents the training data set; is the set of neuron output value vectors of all training samples y in the training data set T; D(x) is the category of the test sample x predicted by the neural network to be tested; K is the Gaussian kernel function, and H is the bandwidth;

[0029] Step 432: Calculate the Euclidean distance between the neuron output value after the single test sample executes the neural network to be tested and the neuron output value after the training sample of the training data set executes the neural network, and obtain the distance-based unexpected sufficiency, which is expressed as:

[0030]

[0031]

[0032] Among them, y a represents the same type of training sample that is closest to the test sample x and belongs to the same category, y b Represents the same training sample y a The closest heterogeneous training samples belonging to different categories; T represents the training data set; T D(x) represents the subset of training samples in the training data set that belong to the same category as the test sample; T / T D(x) Represents a subset of training samples in the training data set that do not belong to the same category as the test samples; Indicates T D(x) The neuron output value vector of the important neuron set in the target layer corresponding to the i-th training sample; Indicates T / T D(x) The neuron output value vector of the important neuron set in the target layer corresponding to the j-th training sample;

[0033] Step 434: Select a target layer in the neural network to be tested, input the training sample y into the neural network, calculate the sum of the correlation scores of all neurons in each network layer, and obtain the output f(y); for each training sample, use the Layer-Wise correlation propagation method to back-propagate the output f(y) layer by layer to obtain the correlation score of each neuron in the target layer, which is expressed as:

[0034]

[0035] in, represents the correlation score of the ith neuron in the lth layer, is the correlation score of the jth neuron in layer l+1; α iis the output value of the i-th neuron; ω represents the connection weight between neurons; h is the total number of neurons in the l-th layer; ε is a small real number to avoid the denominator being too low and close to 0;

[0036] Accumulate the relevance score of each neuron in the target layer on each training sample to obtain the relevance score of each neuron on the training data set;

[0037] The neurons ranked in the top m% of the correlation scores in the target layer are selected as important neurons to form the important neuron set of the target layer;

[0038] Step 435: input the test sample into the neural network to be tested, calculate the likelihood-based surprise sufficiency and the distance-based surprise sufficiency according to the neuron output values ​​corresponding to the important neuron set of the target layer, and calculate according to the likelihood-based surprise sufficiency and the distance-based surprise sufficiency;

[0039] Determine the upper bound of unexpected sufficiency and the lower bound of unexpected sufficiency according to the test case data set, and divide the interval formed by the upper bound of unexpected sufficiency and the lower bound of unexpected sufficiency into k test intervals;

[0040] If the likelihood-based or distance-based accidental sufficiency of the test sample is within a test interval, the test interval is covered. The accidental coverage rate SC is calculated based on the ratio of the number of covered test intervals to the total number of test intervals. The accidental coverage rate includes the likelihood-based accidental sufficiency coverage rate and the distance-based accidental sufficiency coverage rate. The calculation expressions are:

[0041]

[0042] Where L is the lower bound of the accidental sufficiency of the test case data set; U is the upper bound of the accidental sufficiency of the test case data set; the [L,U] interval is evenly divided into k segments to obtain k test intervals, S i represents the i-th test interval; if the likelihood-based or distance-based accidental sufficiency calculated for the test sample x is within S i If S i is covered; SA(x) represents the likelihood-based unexpected sufficiency or the distance-based unexpected sufficiency; substituting the likelihood-based unexpected sufficiency and the distance-based unexpected sufficiency into the above formula respectively, we can obtain the likelihood-based unexpected sufficiency coverage and the distance-based unexpected sufficiency coverage respectively.

[0043] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a method for evaluating the adequacy of intelligent software testing. It analyzes the intelligent software coverage analysis model from three aspects: neuron coverage, activation path, and accidental coverage, and proposes a test adequacy evaluation method for different coverage criteria, which is helpful to improve the adequacy and efficiency of intelligent software testing, provide a strong technical capability guarantee for carrying out intelligent system verification work, and is conducive to improving the quality and reliability level of intelligent software systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A schematic diagram of an intelligent software testing adequacy evaluation method in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of constructing a sufficiency evaluation index based on neuron coverage provided by the present invention;

[0047] Figure 3 A schematic diagram of constructing the adequacy evaluation index based on neuron path coverage provided by the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The embodiment of the present invention discloses a method for evaluating the adequacy of intelligent software testing, including: proposing an adequacy evaluation index of structured neuron coverage based on neuron activation value from two aspects of neuron activation coverage and neuron network area coverage; proposing indicators such as structure-based neuron path coverage, activation-based neuron path coverage, vulnerable neuron path coverage, and adversarial neuron path coverage based on the idea of ​​activation pathway; quantifying the "surprise" of a single test sample relative to a training set based on the accidental adequacy index, and proposing accidental coverage criteria, including likelihood-based accidental coverage and distance-based accidental coverage indicators. Specifically:

[0050] 1) Adequacy evaluation index based on neuron coverage: Use the coverage standard of deep neural network neurons to measure whether the generated test set can cover a sufficient number of neurons in the model. Currently, there are neuron coverage (NC), K-multi-section neuron coverage (KMNC), neuron boundary coverage (NBC), Top-k neuron coverage (TKNC), etc. However, these coverage indicators still have problems including narrow coverage of neuron coverage, K-multi-section neuron coverage only considering neurons in a certain interval, neuron boundary coverage only considering boundary areas, and the k value of Top-k neuron coverage is difficult to determine. In this embodiment, in the intelligent software testing, neuron activation coverage criteria and neuron network area coverage criteria are proposed respectively. First, based on the structured coverage analysis of neuron activation values, an adequacy evaluation index based on neuron coverage is proposed from the two aspects of neuron activation coverage and neuron network area coverage. The main steps of this method include:

[0051] Step 1: Analyze the activation values ​​of neurons in the neural network system. The activation values ​​play an important role in discovering the boundary behavior of the neural network system.

[0052] Step 2: Obtain the distribution of neuron activation values ​​or the changing relationship of activation values ​​between neurons in adjacent layers by statistics and tracking;

[0053] Step 3: Define the coverage of the test samples for the neural network system and the test input generation method based on coverage guidance; the test cases corresponding to the test case data set include test samples, expected outputs and steps;

[0054] Step 4: In terms of neuron activation coverage, count the specific number of neurons with the highest activation degree in each layer of the neural network and calculate the neuron activation coverage rate;

[0055] Step 5: In terms of neural network area coverage, analyze the main working area of ​​neurons and calculate the neural network area coverage rate.

[0056] 2) Adequacy evaluation index based on activation pathway: The activation pathway method is used to construct a deep neural network adequacy evaluation index, using N = {n1, n2, ...} to represent the neurons in the deep neural network D, and using X = {x1, x2, ...} as a series of inputs, and using α n (x) represents the output of neuron n with respect to input x, for an ordered set of neurons N, and Use α N (x) represents a vector of activation values, which is called an activation path. The dimension of this vector is equal to the number of neurons in the neuron set N, called α N (x) is the activation path of input x on the neuron set N. Similarly, A N(X) represents a series of activation pathways. For each neuron in N, an input set X:A N (X) = {α N (x)|x∈X}, the activation path is available after each network execution.

[0057] Based on the path coverage criterion method in traditional software testing, this embodiment proposes a neuron path coverage (NPC) method, and the extracted critical decision path (CDP) is used to display the logic of the deep neural network (DNN). The specific steps include:

[0058] Step 1: The DNN decision logic is determined by the training data. Given the target DNN, the CDP of each training data in the training dataset is extracted. The extracted CDP represents the decision logic learned from the training data.

[0059] Step 2: Given an input, measure the coverage by calculating the distance (i.e., the difference in decision logic) between the CDP of the given input (test sample) and the CDP of the training data;

[0060] Step 3: Perform intra-class path clustering and abstraction on the CDP of the training samples. For inputs predicted to be of the same class, group their CDPs into k clusters. Since calculating the distance of all training samples is costly, clustering and abstraction are used to improve efficiency. Samples with similar decision logic may have similar CDPs. Grouping CDPs of the same class into one group means that inputs with similar decision logic are clustered into one group.

[0061] Step 4: For each cluster, we further propose path abstraction to merge the input CDPs in the cluster into an abstract CDP; the abstract CDP represents the decision logic of all inputs in the cluster, which significantly reduces the number of paths.

[0062] Step 5: Determine the abstract CDP for a given input and propose two coverage criteria. From the control flow perspective, calculate the similarity between the new CDP and the abstract CDP of the corresponding cluster. From the data flow perspective, consider the activation values ​​of neurons in the abstract CDP and calculate the distance between the input samples and the training samples in the cluster (along the abstract CDP). The distance-based coverage criterion measures the extent to which a new decision logic is covered by a given test component.

[0063] Step 6: Similarity and distance reveal how the decision behavior of new inputs is different. Divide the similarity and distance into m parts and calculate how many parts are covered by a given test component. Dividing into m parts means eliminating duplicate data in all similarities and distances. Count the number of similarities and distances as m. Calculate how many parts are covered, that is, calculate the ratio of m to the number of CDPs in step 2. This ratio is used as the neuron path coverage.

[0064] 3) Surprise coverage-based sufficiency evaluation indicators: A fine-grained test sufficiency framework SADL for deep neural network testing is proposed to calculate the "surprise value" of the test input relative to the training input, and introduces two indicators: surprise adequacy (SA) and surprise coverage (SC). SADL provides two methods for calculating SA based on kernel density estimation (KDE) and Euclidean distance. By comparing the difference in SA values ​​between the test input and the training data set, the surprise value of the test input relative to the training data is expressed. SA and SC can accurately capture the unexpected value of the input, which is important for distinguishing adversarial samples and guiding model retraining. It solves the problem that the existing deep neural network test coverage standards are not fine-grained and there is little analysis of single test input information.

[0065] This embodiment proposes an intelligent software test coverage index based on improved unexpected sufficiency, uses the unexpected sufficiency index to quantify the "unexpectedness" of a single test sample relative to the training set, and designs an unexpected coverage criterion on this basis to measure the sufficiency of intelligent software testing. The main steps of the method include:

[0066] Step 1: Construct test coverage criteria based on surprise adequacy; surprise adequacy is an indicator for a single test sample, which is used to quantify the difference between the test sample and the training set in the output of neurons in the DNN model. The lower the surprise adequacy value, the more similar the neuron output of the test sample is to the training set. The higher the value, the greater the difference between the neuron output value of the test sample and the training set, which is more likely to reveal hidden errors in the DNN model. Surprise adequacy and surprise coverage introduce the probability of surprise adequacy from a non-structural perspective, and propose likelihood-based surprise adequacy (LSA) and distance-based surprise adequacy (DSA), on which the surprise coverage of the test set is measured. First, given a test sample x, Represents the vector composed of the output values ​​of all neurons in the lth layer of the DNN model, is the set of neuron output value vectors of all samples in the training data set T; D(x) is the category of the test sample x predicted by the model, T D(x) It represents the subset of training samples that belong to the same category as the test sample. The unexpected sufficiency metric is described as follows:

[0067] 1) Likelihood-based Accidental Sufficiency: Given a test sample x, this metric uses kernel density estimation to estimate the likelihood of similar samples appearing in the DNN model D during training, which is defined as follows:

[0068]

[0069] Among them, K is the Gaussian kernel function, H is the bandwidth; when the neural network to be tested is a classification model, T can be used D(x) Replacing T in the formula, this indicator shows that the probability density function value of the test sample that is more different from the training set is smaller.

[0070] 2) Distance-based accidental sufficiency: This indicator quantifies the difference between samples based on the Euclidean distance between the neuron output vector of a single test sample and the neuron output vector of the training sample. This indicator shows that the closer the test sample is to the decision boundary between the two classes, the greater the difference between the test sample and the training set. Since the DSA indicator uses the distance between samples to characterize the decision boundary between DNN model categories, this indicator is only applicable to classification models, which is expressed as follows:

[0071]

[0072] where y a represents the training sample that is closest to x and belongs to the same category, y b Represents y a The closest training samples that belong to different categories;

[0073] Step 2: Screening of important neurons in the neural network; select a specific hidden layer in the network as the target layer. This embodiment uses the Layer-Wise correlation propagation method (LRP algorithm), inputs a data set, uses the neuron forward output value and the weight between neurons, measures the correlation score between each neuron in the DNN model and the output result, obtains the actual contribution of each neuron in the network to the decision, and completes the identification of important neurons in the target layer. Specifically:

[0074] Input the training sample y into the DNN model and obtain the output f(y). The sum of the correlation scores of all neurons in each network layer is equal to f(y). For each training sample, the LRP algorithm back-propagates the output of the model layer by layer and calculates the correlation score of each neuron in the target layer according to formula (3).

[0075]

[0076] in represents the correlation score of the ith neuron in the lth layer, is the correlation score of the jth neuron in layer l+1; α iis the output value of the ith neuron, ω represents the connection weight between neurons, h is the total number of neurons in the lth layer, and ε is a small real number to avoid the denominator being too low and close to 0. It can be seen from formula (3) that the higher the output value of the neuron, the greater the impact on the decision result; the larger the connection weight, the higher the correlation between neurons.

[0077] For each neuron in the target layer, the correlation scores calculated on each training sample are accumulated to obtain the correlation score of each neuron in the training set; the neurons with the top m% correlation scores in each layer are selected as important neurons to form the set of important neurons in the target layer, which is expressed as For different DNN models, the optimal value of m can be found through experiments.

[0078] Step 3: Calculation of sufficiency evaluation index based on accidental coverage: Given a test set, L is the lower bound of the accidental sufficiency (SA value) of the test set, U is the upper bound of the preset SA value, and the [L, U] interval is evenly divided into k segments, S i represents the i-th segment; if the DSA or LSA value calculated for the test sample x is in segment S i If the test set is within the range of , the segment is covered. The accidental coverage is calculated as the proportion of the segment covered by the test set. The specific calculation formula is as follows:

[0079]

[0080] SA(x) represents likelihood-based surprise sufficiency or distance-based surprise sufficiency; Substituting likelihood-based surprise sufficiency and distance-based surprise sufficiency into the above formula respectively, we can obtain likelihood-based surprise sufficiency coverage and distance-based surprise sufficiency coverage respectively; The higher the corresponding surprise coverage of the test set, the more diverse the test set is, which contains both inputs similar to the training data and inputs different from the training data; DSC and LSC represent the surprise coverage criteria measured based on the DSA indicator and the LSA indicator respectively.

[0081] The unexpected sufficiency evaluation index uses the intermediate representation generated by the DNN model, that is, the neuron output value of the sample, to quantify the difference between a single test sample and the training set, and can find the test sample that causes the model to produce incorrect prediction behavior; the output value of the important neuron in the target layer is used as a feature, and the unexpected sufficiency evaluation index is used to quantify the difference between a single test sample and the training set. Substituting N in the above formulas (1) and (2) l , using the vector composed of the output values ​​corresponding to the important neuron set in the target layer of the DNN model The LSA and DSA values ​​of the test samples are measured; the two unexpected sufficiency metrics are expressed as LSA v and DSC v , the corresponding accidental coverage index is expressed as LSCv and DSC v ;

[0082] The formula after substitution is:

[0083]

[0084] in, Represents a neuron output value vector consisting of neuron output values ​​corresponding to an important neuron set in a target layer of a neural network to be tested; is the set of neuron output value vectors of all training samples y in the training data set T; Indicates T D(x) The neuron output value vector of the important neuron set in the target layer corresponding to the i-th training sample; Indicates T / T D(x) The neuron output value vector of the important neuron set in the target layer corresponding to the j-th training sample.

[0085] On the other hand, in a specific embodiment, for the image target detection YOLOv5 intelligent algorithm, the coverage analysis and application verification are carried out according to the intelligent software test adequacy evaluation method proposed in the present invention. The specific implementation plan is as follows Figure 1 As shown, the specific steps are as follows:

[0086] S1: For the target detection algorithm, based on the original dataset (including COCO dataset and self-collected dataset), a test case dataset is generated, referred to as the test dataset; at the same time, a training dataset is constructed;

[0087] S2: Design and build intelligent software testing environment, including target detection intelligent algorithm training environment and intelligent algorithm testing environment, and conduct neural network coverage testing of intelligent algorithms on training data sets and test data sets;

[0088] S3: Perform target detection YOLOv5 neural network model training and testing; in the intelligent algorithm training environment, perform image target detection intelligent algorithm training and debugging, and complete algorithm convergence, efficiency, and correctness testing; in the intelligent algorithm testing environment, import the image target detection intelligent algorithm model file (the relevant files of the trained target detection YOLOv5 neural network model) and the test data set into the intelligent algorithm testing and evaluation platform, perform neuron coverage analysis, extract data key decision paths, and calculate important neuron output variables, and correspondingly implement neuron activation and area coverage analysis, neuron path coverage analysis, and accidental coverage analysis, and complete the algorithm coverage test;

[0089] S4: Obtain various coverage data through statistics. It is necessary to use the test evaluation platform tool to perform coverage analysis to obtain neuron coverage data, path coverage data and accidental coverage data, calculate neuron coverage, neuron path coverage and accidental coverage data, and complete the test adequacy evaluation.

[0090] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0091] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An intelligent software testing adequacy evaluation method, characterized in that: The following steps are involved: Step 1: Build test case dataset and training dataset; Step 2: Build an intelligent software testing environment, including a neural network training environment and a neural network testing environment; Step 3: Train the neural network of the intelligent software in the neural network training environment using the training data set to obtain the neural network to be tested; Step 4: In the neural network test environment, use the test case data set to perform neuron coverage test, path coverage test and accidental coverage test on the neural network to be tested, and obtain neuron coverage rate, neuron path coverage rate and accidental coverage rate respectively; Step 5: Compare and judge based on neuron coverage, neuron path coverage, unexpected coverage and the set coverage threshold to obtain the adequacy evaluation result.

2. The intelligent software testing adequacy evaluation method according to claim 1, characterized in that: When performing neuron coverage testing in step 4, a structured coverage analysis based on neuron activation values ​​is performed according to the neuron activation coverage criteria and the neuron network area coverage criteria to obtain neuron coverage, which includes neuron activation coverage and neuron network area coverage. The specific process is: Step 411: input the test samples in the test case data set into the neural network to be tested, and obtain the neuron activation value of the neural network to be tested; Step 412: Calculate the neuron activation value distribution or the change of the neuron activation value of the adjacent layer according to the neuron activation value; Step 413: according to the distribution of neuron activation values, changes in the activation values ​​of neurons in adjacent layers and the set activation threshold, count the activated neurons in each layer of the network structure in the neural network, calculate the ratio of the number of activated neurons to the number of all neurons in the neural network to be tested, and obtain the neuron activation coverage; Step 414: Input the test samples in the test case data set into the neural network to be tested, obtain the executed neuron working area, calculate the ratio of the executed neuron working area to the total neuron working area in the neural network to be tested, and obtain the neuron network area coverage.

3. The intelligent software testing adequacy evaluation method according to claim 1, characterized in that: The specific process of performing path coverage test in step 4 to obtain neuron path coverage is as follows: Step 421: extracting the key decision path of training for each training sample in the training data set of the neural network training process in step 3 respectively; Step 422: extracting the key decision path of each test sample in the test case data set after inputting it into the neural network to be tested for execution; Step 423: clustering and abstracting the training key decision paths of all training samples into k clusters; Step 424: abstract paths for all clusters, and merge the key decision paths in each cluster into an abstract key decision path; Step 425: Calculate the similarity between the test key decision path in step 422 and the abstract key decision path of each cluster, and calculate the distance between the test sample and the training sample; Step 426: Eliminate duplicate data in all similarities and distances, count the number of similarities and distances, obtain the path coverage value, calculate the ratio of the path coverage value to the number of key decision paths tested, and obtain the neuron path coverage rate.

4. The intelligent software testing adequacy evaluation method according to claim 1, characterized in that: The specific process of performing the accidental coverage test in step 4 to obtain the accidental coverage rate is as follows: Step 431: using a kernel density estimation method to calculate likelihood-based surprise sufficiency of test samples in the test case data set; Step 432: Calculate the Euclidean distance between the neuron output value after the single test sample executes the neural network to be tested and the neuron output value after the training sample of the training data set executes the neural network, and obtain the distance-based unexpected sufficiency; Step 434: select a target layer in the neural network to be tested, input the training sample x into the neural network, calculate the correlation score using the Layer-Wise correlation propagation method, and select the important neuron set of the target layer according to the correlation score; Step 435: Input the test sample into the neural network to be tested, calculate the likelihood-based surprise sufficiency and the distance-based surprise sufficiency according to the neuron output values ​​corresponding to the important neuron set of the target layer, and calculate the surprise coverage rate according to the likelihood-based surprise sufficiency and the distance-based surprise sufficiency.

5. The intelligent software testing adequacy evaluation method according to claim 4, characterized in that: The expression for calculating likelihood-based surprise sufficiency LSA is: Where x represents the test sample; represents the neuron output value vector composed of the neuron output values ​​corresponding to the important neuron set of the target layer in the neural network to be tested; T represents the training data set; is the set of neuron output value vectors of all training samples y in the training data set T; D(x) is the category of the test sample x predicted by the neural network to be tested; K is the Gaussian kernel function, and H is the bandwidth.

6. The intelligent software testing adequacy evaluation method according to claim 4, characterized in that: The expression for calculating the distance-based accidental sufficiency DSA is: Where x represents the test sample; Represents the neuron output value vector composed of the neuron output values ​​corresponding to the important neuron set of the target layer in the neural network to be tested; a represents the same type of training sample that is closest to the test sample x and belongs to the same category, y b Represents the same training sample y a The closest heterogeneous training samples belonging to different categories; T represents the training data set; T D (x) represents the subset of training samples in the training data set that belongs to the same category as the test sample; T / T D (x) represents the subset of training samples in the training data set that do not belong to the same category as the test samples; Indicates T D (x) The neuron output value vector of the important neuron set in the target layer corresponding to the i-th training sample; Indicates T / T D The neuron output value vector of the important neuron set in the target layer corresponding to the j-th training sample in (x).

7. The intelligent software testing adequacy evaluation method according to claim 4, characterized in that: The specific process of step 434 is: Step 4341: Select a target layer in the neural network, input the training sample y into the neural network, calculate the sum of the correlation scores of all neurons in each network layer, and obtain the output f(y); Step 4342: For each training sample, the output f(y) is back-propagated layer by layer using the Layer-Wise correlation propagation method to obtain the correlation score of each neuron in the target layer, which is expressed as: in, represents the correlation score of the ith neuron in the lth layer, is the correlation score of the jth neuron in layer l+1; α i is the output value of the i-th neuron; ω represents the connection weight between neurons; h is the total number of neurons in the l-th layer; ε is a set real number; Step 4343: Accumulate the correlation score of each neuron in the target layer on each training sample to obtain the correlation score of each neuron on the training data set; Step 4344: Select the neurons in the target layer whose correlation scores are ranked in the top m% as important neurons to form the important neuron set of the target layer.

8. The intelligent software testing adequacy evaluation method according to claim 4, characterized in that: Determine the upper bound of unexpected sufficiency and the lower bound of unexpected sufficiency according to the test case data set, and divide the interval formed by the upper bound of unexpected sufficiency and the lower bound of unexpected sufficiency into k test intervals; If the likelihood-based or distance-based accidental sufficiency of the test sample is within a test interval, the test interval is covered. The accidental coverage rate SC is calculated based on the ratio of the number of covered test intervals to the total number of test intervals. The accidental coverage rate includes the likelihood-based accidental sufficiency coverage rate and the distance-based accidental sufficiency coverage rate. The calculation expressions are: Where L is the lower bound of the accidental sufficiency of the test case data set; U is the upper bound of the accidental sufficiency of the test case data set; the [L,U] interval is evenly divided into k segments to obtain k test intervals, S i Indicates the i-th test interval; If the likelihood-based or distance-based surprise sufficiency calculated for the test sample x is within S i If S i is covered; SA(x) represents likelihood-based surprise sufficiency or distance-based surprise sufficiency.

Images