An ahp and bpnn-based software testing organization comprehensive evaluation method and system

Abstract

The application relates to a software testing organization comprehensive evaluation method and system based on AHP and BPNN, wherein the method comprises the following steps: constructing a testing organization evaluation index system and a testing project evaluation index system based on a testing organization evaluation index and a testing project evaluation index respectively; establishing a testing organization basic evaluation model by using the analytic hierarchy process based on the testing organization evaluation index; establishing and training a testing project basic evaluation model by using the analytic hierarchy process and BPNN based on the testing project evaluation index; comprehensively obtaining a comprehensive evaluation model by using the analytic hierarchy process to comprehensively obtain the testing organization basic evaluation model and a preselected number of testing project average evaluation models; and collecting the testing organization evaluation index and the testing project evaluation index and inputting the testing organization evaluation index and the testing project evaluation index into the comprehensive evaluation model to obtain an evaluation result. Through the application, the software testing organization can be comprehensively and quickly evaluated, and data support can be provided for task allocation decision-making of an evaluation task assigner.

Description

Technical Field

[0001] This application relates to the field of software testing and evaluation technology, and in particular to a comprehensive evaluation method and system for software testing organizations based on AHP and BPNN. Background Technology

[0002] With the rapid development of science and technology, computer software is being used more and more widely, improving work efficiency in all walks of life. However, software malfunctions can lead to major security incidents. For example, on March 7, 2021, the Hubble Space Telescope entered safe mode due to an onboard software error, suspending all scientific operations. Ultimately, this was caused by insufficient software testing.

[0003] Different software testing organizations have varying testing capabilities. Even within the same organization, the adequacy of testing for a given project can fluctuate due to differences in tester skills and work schedules. Many factors influence the evaluation of software testing organizations. A common approach is to intuitively select an organization based on its accreditations and certifications, relying on test outlines and expert review meetings to qualitatively assess the software testing results. This method is insufficient for accurately assessing the organization's capabilities and is easily influenced by human factors.

[0004] Therefore, no effective solution has yet been proposed for comprehensively evaluating the testing capabilities of software testing organizations based on the testing projects they have completed, so as to effectively, quickly and economically select software testing organizations. Summary of the Invention

[0005] This application provides a comprehensive evaluation method and system for software testing organizations based on AHP and BPNN. It utilizes the Analytic Hierarchy Process (AHP) and BPNN (Back Propagation Neural Network) and combines them with evaluation organizations and testing projects to provide evaluation indicators and evaluation models. This aims to at least solve the problem of how to comprehensively and quickly evaluate software testing organizations and provide data support for the task allocation decision of the task assignor.

[0006] In a first aspect, embodiments of this application provide a comprehensive evaluation method for software testing organizations based on AHP and BPNN, including:

[0007] The evaluation index system is constructed by building an evaluation index system for testing organizations and an evaluation index system for testing projects, based on a set of evaluation indexes for testing organizations and a set of evaluation indexes for testing projects, respectively. Specifically, the evaluation indexes for testing organizations include: qualifications, personnel capabilities, testing quality, contract progress, price, service quality, ability to accept urgent tasks, testing tools, and external environment. The evaluation index system for testing projects includes: test type, test case coverage, requirement coverage, test coverage, average code execution time, and average test case execution time.

[0008] The steps for establishing a basic evaluation model for testing organizations are as follows: Based on the evaluation indicators of testing organizations, a basic evaluation model for testing organizations is established using the analytic hierarchy process.

[0009] The steps for establishing a basic evaluation model for test items are as follows: Based on the evaluation indicators of the test items, a basic evaluation model for test items is established and trained using the analytic hierarchy process and BPNN.

[0010] The steps for establishing a comprehensive evaluation model are as follows: The comprehensive evaluation model is obtained by combining the basic evaluation model of the testing organization and the average evaluation model of a pre-selected number of test items using the analytic hierarchy process. When a portion of the test items are selected to evaluate the testing organization, the basic evaluation model of the test items is the average evaluation model of the test items.

[0011] The evaluation result acquisition steps involve collecting the evaluation indicators of the testing organization and the evaluation indicators of the testing items, inputting them into the comprehensive evaluation model, and calculating the evaluation result P. Z .

[0012] In some embodiments, the comprehensive evaluation model is represented as the following calculation model:

[0013]

[0014] in, The evaluation result of the basic evaluation model of the testing organization is w1. The weight, w2 is the evaluation result of the average evaluation model for the test items. The weight.

[0015] In some embodiments, the evaluation results of the average evaluation model for the test items are... The following calculation model was used to obtain the following:

[0016]

[0017] Where N is the number of selected test items, and P i The evaluation results are for the basic evaluation model of the test item.

[0018] In some embodiments, the steps for establishing a basic evaluation model for the testing organization further include:

[0019] The judgment matrix construction step involves constructing judgment matrix A based on the relative importance of the evaluation indicators of the testing organization. jud =(a ij ) n×n ;

[0020] The consistency check step involves calculating the largest eigenvalue λ of the judgment matrix. max After obtaining the corresponding feature vector S, calculate the consistency ratio CR. If the consistency ratio CR < 0.1, the judgment matrix passes the consistency test; otherwise, return to the judgment matrix construction step for correction.

[0021] The steps for obtaining the indicator weight vector are as follows: normalize the feature vector S to obtain the indicator weight vector ω.

[0022] The steps for determining the basic evaluation model of a testing organization include weighted summation of the testing organization evaluation indicators and the indicator weight vector ω to obtain the basic evaluation model of the testing organization, which is expressed as follows:

[0023]

[0024] In the formula, n is the number of evaluation indicators, x i Let be the value of the i-th indicator.

[0025] In some embodiments, the consistency ratio is calculated based on the following calculation model:

[0026] CR = CI / RI

[0027] Wherein, the consistency index CI = (λ max -n) / (n-1), the average random consistency index RI is obtained by looking up a table, which has a pre-configured RI value corresponding to n.

[0028] In some embodiments, the step of establishing the basic evaluation model for the test item further includes:

[0029] The steps for building a BP neural network model include configuring BP neural network parameters and constructing a three-layer BP neural network structure model. The specific BP neural network parameters include: the number of neurons in the input layer, the number of neurons in the output layer, the number of neurons in the hidden layer, the activation function, the learning rate, and the termination condition.

[0030] The training sample construction steps are as follows: select a set of test items, evaluate the test items using the analytic hierarchy process based on the evaluation indicators of the test items, and use the input values ​​of the indicators of all test items and the corresponding evaluation results as training samples.

[0031] The network model training steps involve inputting the training samples into the BP neural network model for training to obtain the weights of each neuron, where the weight matrix from the input layer to the hidden layer is W, and the weight vector from the hidden layer to the output layer is V.

[0032] The steps for determining the basic evaluation model of the test item are as follows: the basic evaluation model of the test item is determined based on the weight matrix W and the weight vector V.

[0033] In some embodiments, the basic evaluation model for the test item can be calculated using the following computational model:

[0034]

[0035] In the formula, P represents the basic evaluation result of the test item, n represents the number of neurons in the input layer, m represents the number of neurons in the hidden layer, and x represents the number of neurons in the hidden layer. i w represents the actual score of the i-th evaluation metric in the input layer. ij Let v be the weights from the i-th index in the input layer to the j-th neuron in the hidden layer. j Let f(·) be the weight from the j-th neuron in the hidden layer to the neuron in the output layer, and f(·) be the neuron activation function.

[0036] Secondly, embodiments of this application provide a comprehensive evaluation system for software testing organizations based on AHP and BPNN, used to implement the comprehensive evaluation method for software testing organizations based on AHP and BPNN as described in the first aspect above. The system includes:

[0037] The evaluation index system construction module is used to construct an evaluation index system for testing organizations and an evaluation index system for testing projects based on an evaluation index for a testing organization and an evaluation index for a testing project, respectively.

[0038] The module for establishing a basic evaluation model for testing organizations is used to establish a basic evaluation model for testing organizations based on the evaluation indicators of the testing organizations using the analytic hierarchy process.

[0039] The test project basic evaluation model establishment module is used to establish and train a test project basic evaluation model based on the test project evaluation indicators using the analytic hierarchy process and BPNN.

[0040] The comprehensive evaluation model building module is used to combine the basic evaluation model of the testing organization and the average evaluation model of a pre-selected number of test items using the analytic hierarchy process to obtain a comprehensive evaluation model. When a portion of the test items are selected to evaluate the testing organization, the basic evaluation model of the test items is the average evaluation model of the test items.

[0041] The evaluation result acquisition module is used to collect the evaluation indicators of the testing organization and the evaluation indicators of the testing items, input them into the comprehensive evaluation model, and calculate the evaluation result P. Z .

[0042] In some embodiments, the testing facility basic evaluation model establishment module further includes:

[0043] The judgment matrix construction module is used to construct a judgment matrix A based on the relative importance of the evaluation indicators of the testing organization. jud =(a ij ) n×n ;

[0044] The consistency verification module is used to calculate the largest eigenvalue λ of the judgment matrix. max After determining the corresponding eigenvector S, calculate the consistency ratio CR. If the consistency ratio CR < 0.1, then the judgment matrix passes the consistency test.

[0045] The indicator weight vector acquisition module is used to normalize the feature vector S to obtain the indicator weight vector ω.

[0046] The testing organization basic evaluation model determination module is used to obtain the testing organization basic evaluation model by weighted summation of the testing organization evaluation indicators and the indicator weight vector ω.

[0047] In some embodiments, the test item basic evaluation model building module further includes:

[0048] The BP neural network model building module is used to configure BP neural network parameters and build a three-layer BP neural network structure model;

[0049] The training sample construction module is used to select a set of test items, evaluate the test items using the analytic hierarchy process based on the evaluation indicators of the test items, and use the input values ​​of the indicators of all test items and the corresponding evaluation results as training samples.

[0050] The network model training module is used to input the training samples into the BP neural network model for training and obtain the weights of each neuron, wherein the weight matrix from the input layer to the hidden layer is W, and the weight vector from the hidden layer to the output layer is V.

[0051] The test item basic evaluation model determination module is used to determine the test item basic evaluation model based on the weight matrix W and the weight vector V.

[0052] Compared to related technologies, the comprehensive evaluation method and system for software testing organizations based on AHP and BPNN provided in this application evaluates testing organizations from both the perspectives of testing organizations and testing projects by establishing a basic evaluation model for testing organizations and a basic evaluation model for testing projects. It also clarifies the corresponding evaluation indicators and the measurement methods for the values ​​of the evaluation indicators. This can effectively, quickly, and economically evaluate testing organizations and provides important guidance for the task allocation decisions of the task assignor. It is simple to implement, convenient, and fast. At the same time, when the evaluation indicators change, the model can be quickly updated to obtain a new evaluation model. It is easy to modify and has a wide range of applications.

[0053] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is a flowchart of the comprehensive evaluation method according to an embodiment of this application;

[0056] Figure 2 This is a step-by-step flowchart of the comprehensive evaluation method according to an embodiment of this application;

[0057] Figure 3 This is another flowchart of the comprehensive evaluation method according to an embodiment of this application;

[0058] Figure 4 Flowchart of the BP neural network model training algorithm;

[0059] Figure 5 This is a structural block diagram of a comprehensive evaluation system according to an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0061] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0062] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0063] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0064] The evaluation of software testing organizations is influenced by numerous factors. Conventional methods rely on intuition, based on various accreditations and certifications, and qualitatively assess the results of software testing through test outlines and expert review meetings. This approach is insufficient for accurately assessing the organization's testing capabilities and is easily influenced by human factors. To address this issue, this application provides a comprehensive evaluation method for software testing organizations based on AHP and BPNN, employing both qualitative and quantitative methods. Figure 1-3 This is a flowchart of the comprehensive evaluation method according to an embodiment of this application, with reference to... Figure 1-3 As shown, the process includes the following steps:

[0065] Step S1 of the evaluation index system construction involves constructing an evaluation index system for testing organizations and an evaluation index system for testing projects based on a set of evaluation indexes for testing organizations and a set of evaluation indexes for testing projects, respectively. Specifically, the evaluation indexes for testing organizations include: qualifications, personnel capabilities, test quality, contract progress, price, service quality, ability to accept urgent tasks, testing tools, and external environment. The evaluation index system for testing projects includes: test type, test case coverage, requirement coverage, test coverage, average code execution time, and average test case execution time.

[0066] Step S2: Establish a basic evaluation model for testing organizations based on the evaluation indicators of testing organizations and using the analytic hierarchy process.

[0067] Step S3: Establishing a basic evaluation model for test items based on test item evaluation indicators using the analytic hierarchy process and BPNN to build and train a basic evaluation model for test items.

[0068] Step S4 of the comprehensive evaluation model establishment involves using the analytic hierarchy process (AHP) to combine the basic evaluation model of the testing organization and the average evaluation model of the pre-selected number of test items to obtain the comprehensive evaluation model. Considering that the number of test items undertaken by the testing organization is large and cannot all be included in the evaluation, when a portion of the test items are selected to evaluate the testing organization, the basic evaluation model of the test items is the average evaluation model of the test items.

[0069] In step S5, the evaluation results are obtained by collecting the evaluation indicators of the testing organization and the evaluation indicators of the testing items and inputting them into the comprehensive evaluation model to calculate the evaluation result P. Z .

[0070] In some embodiments, the comprehensive evaluation model is represented by the following calculation model:

[0071]

[0072] in, The evaluation results of the testing organization's basic evaluation model, w1 is... The weight, w2 represents the evaluation result of the average evaluation model for the test items. The weight.

[0073] Among them, the evaluation results of the average evaluation model for the test items The following calculation model was used to obtain the following:

[0074]

[0075] Where N is the number of test items selected, and here N = 10 test items are selected for evaluation, P i To test the evaluation results of the basic evaluation model for the project, then

[0076] In the above embodiments, step S2 of establishing the basic evaluation model for the testing organization in this application further includes:

[0077] Step S201: Construct the judgment matrix A based on the relative importance of the evaluation indicators of the testing organization. jud =(a ij ) n×n The relative importance scale ranges from [1,9], a ij =1 / a ji ,a ii =1, in this step, n is the number of evaluation indicators of the testing organization; the judgment matrix A in this embodiment of the application jud Example as follows:

[0078]

[0079] In the consistency check step S202, the largest eigenvalue λ of the judgment matrix is ​​calculated. max After obtaining the corresponding eigenvector S, the consistency ratio CR is calculated. If the consistency ratio CR < 0.1, the judgment matrix passes the consistency test, and the eigenvector S is valid. Otherwise, the process returns to step S201 of the judgment matrix construction for correction. Based on matrix operations, the largest eigenvalue obtained in step S202 is λ. max =9.7342, S=-0.68333005,-0.51555299,-0.32084125,-0.10674046,- 0.05616799,-0.21428235,-0.16863942,-0.26713111,-0.06373749] T .

[0080] The consistency ratio is calculated based on the following model:

[0081] CR = CI / RI

[0082] Wherein, the consistency index CI = (λ max -n) / (n-1)=0.09177917341513031, the average random consistency index RI is obtained by looking up the table, and the table below has RI values ​​corresponding to n pre-configured.

[0083] RI 0 0 0.52 0.89 1.12 1.26 1.36 1.41 n 9 10 11 12 13 14 15 RI 1.46 1.49 1.52 1.54 1.56 1.58 1.59

[0084] After consulting the table, n is 9, so RI = 1.46, and CR = CI / RI = 0.0628624475446098 < 0.1. In other words, the judgment matrix calculated above passes the consistency test, and its eigenvector S is valid.

[0085] Step S203, obtaining the indicator weight vector, involves normalizing the feature vector S to obtain the indicator weight vector ω. Specifically:

[0086] ω=[0.28514583,0.21513438,0.13388339,0.04454157,0.02343826,0.08941758,0.0703713,0.11147076,0.02659692] T This can be used as an indicator weight vector.

[0087] Step S204 of determining the basic evaluation model for the testing organization involves weighted summation of the testing organization evaluation indicators and the indicator weight vector ω to obtain the basic evaluation model for the testing organization. The basic evaluation model for the testing organization is expressed as follows:

[0088]

[0089] In the formula, n is the number of evaluation indicators, x i Let be the value of the i-th indicator.

[0090] Step S3, which establishes the basic evaluation model for the test project, further includes:

[0091] Step S301 of BP neural network model construction: Configure BP neural network parameters and construct a three-layer BP neural network structure model. BP neural network parameters specifically include: the number of input layer neurons, the number of output layer neurons, the number of hidden layer neurons, the activation function, the learning rate, and the termination condition. Optionally, the number of input layer neurons is configured to be the number of evaluation indicators n for the test items, where n is 6, and the input value range for each indicator is [0,1]. Since the evaluation objective is to obtain one evaluation value through the evaluation model, the number of output layer neurons l is 1, and the output value range is [0,1]. Optionally, based on empirical formulas... The number of hidden layer neurons, m, is set, where a is an adjustment constant between 1 and 10. Assuming a is 3, then m is approximately 6. Optional activation functions include, but are not limited to, the ReLU function, the Sigmoid function, the bisigmoid function, and the tanh function. This application preferably uses the Sigmoid function, i.e.:

[0092]

[0093] Considering that a smaller learning rate requires more training iterations, while an excessively large learning rate can negatively impact the stability of the BP neural network structure, the learning rate in this embodiment is configured between 0.01 and 0.9, preferably 0.21. The termination condition is set so that the sum of squared errors (MSE) of the BP neural network is less than a preset termination threshold, which is set to ε = 1e -4 ;

[0094] In the training sample construction step S302, a set of test items is selected, preferably 100, and the test items are evaluated using the analytic hierarchy process based on the evaluation indicators of the test items. The input values ​​of the indicators of all test items and the corresponding evaluation results are used as training samples.

[0095] Network model training step S303, refer to Figure 4 As shown, training samples are input into the BP neural network model for training, and the weights of each neuron are obtained. The weight matrix from the input layer to the hidden layer is W = (w ij ) n×m The weight vector from the hidden layer to the output layer is V = (v j ) m×1 In this step, n is the number of neurons in the input layer, and m is the number of neurons in the hidden layer; W is configured as follows:

[0096]

[0097] V is configured as follows:

[0098] V=[-4.4105,-3.6685,-5.0383,2.4410,-4.6736,1.0872] T .

[0099] Step S304: Determine the basic evaluation model for the test project based on the weight matrix W and the weight vector V.

[0100] The basic evaluation model for the test items can be calculated using the following calculation model:

[0101]

[0102] In the formula, P represents the basic evaluation result of the test item, n represents the number of neurons in the input layer, m represents the number of neurons in the hidden layer, and x represents the number of neurons in the hidden layer. i w represents the actual score of the i-th evaluation metric in the input layer. ij Let v be the weights from the i-th index in the input layer to the j-th neuron in the hidden layer. j Let f(·) be the weight from the j-th neuron in the hidden layer to the neuron in the output layer, and f(·) be the neuron activation function.

[0103] As in the above embodiment, qualifications are determined through a pre-defined qualification set {a}. i |i=1,2,…N A} and the pre-defined importance of qualifications w ai Calculations show that the preset qualification set can be flexibly adjusted and selected by evaluation experts; the value of qualification A can be calculated using the following calculation model:

[0104] in,

[0105] in,

[0106] As in the above embodiment, personnel capability is calculated by collecting the number of test personnel with titles such as researcher, engineer, and senior engineer in the testing institution and calculating the average personnel title. The value of personnel capability B can be calculated using the following calculation model:

[0107]

[0108] Among them, w bi The capability weights corresponding to different professional titles can be flexibly adjusted by the evaluation experts. b1 is the number of testers with the professional title of researcher, b2 is the number of testers with the professional title of engineer, and b3 is the number of testers with the professional title of senior engineer. It should be noted that although the embodiments of this application select research institute, engineer, and senior engineer as the capability evaluation parameters for testers, if other professional titles or other qualification types are added to software testers as society changes, these parameters will also fall within the protection scope of the embodiments of this application.

[0109] As in the above embodiment, the test quality is calculated by collecting the number of test items and the number of batches accepted for each test item, and calculating the average acceptance rate of the test items. The value of the test quality C can be calculated using the following calculation model:

[0110]

[0111] Where, N C c represents the number of items in the test work. i This refers to batches that have passed the acceptance test.

[0112] As in the above embodiment, the contract progress is based on the number of contracts N. D Number of contracts fulfilled (n) d1 and the number of contracts fulfilled on time n d2 The contract progress D can be calculated using the following model:

[0113]

[0114] in, d1 is the contract fulfillment rate, d2 is the on-time contract fulfillment rate, and w di The preset weights for contract fulfillment rate or on-time contract fulfillment rate can be flexibly adjusted by the evaluation experts.

[0115] As in the above embodiment, the price E is obtained by judging the quotation of the testing agency based on a preset price range and its corresponding score. In this embodiment of the application, the price range and corresponding score are configured such that the price and the score are inversely proportional, that is, the higher the price, the lower the score.

[0116] As in the above embodiment, service quality is determined based on the testing organization's communication and feedback capabilities, service improvement capabilities, and their respective weights. fi The service quality F can be calculated using the following model:

[0117]

[0118] Here, f1 is the score for communication and feedback ability assigned to the evaluation experts, and f2 is the score for service improvement ability assigned to the evaluation experts.

[0119] As in the above embodiment, the indicator of the ability to accept emergency tasks is obtained by collecting the historical number N of emergency tasks from the testing organization. G Number of urgent tasks completed (n) g Furthermore, it was calculated that the value of the emergency task acceptance capability G can be obtained through the following calculation model:

[0120]

[0121] As in the above embodiment, the average effectiveness of the testing tools in the testing organization is calculated by collecting the types of testing tools and the number of tools of each type. The value of the testing tool H can be calculated using the following calculation model:

[0122]

[0123] Where, N H For test tool type, (h i ,w hi ), i = 1, 2, ..., N HThe number of tools for each type of testing tool and their preset weights are specified, and these weights can be flexibly adjusted by the evaluation experts.

[0124] As in the above embodiment, the external environment is measured based on the geographical distance between the testing organization and the party issuing the testing task. In this embodiment, the closer the testing organization is to the party issuing the testing task, the higher the score. Therefore, the value of external environment I is the score corresponding to the distance interval between the testing organization and the party issuing the testing task.

[0125] As in the above embodiment, the test types further include unit testing, component testing, configuration item testing, and system testing. The value of test type J can be calculated using the following calculation model:

[0126] in

[0127] Where, j i For each test type, i = 1, 2, 3, 4, w ji Preset weights are assigned to various test types, and these weights can be flexibly adjusted by evaluation experts.

[0128] As in the above embodiment, the test case coverage is calculated based on the number of test cases T and the line of code (LOC). The test case coverage K can be calculated using the following model:

[0129]

[0130] As in the above embodiment, the requirement coverage rate is calculated based on the number of requirements covered by test cases r and the total number of requirements R. The requirement coverage rate L can be calculated using the following calculation model:

[0131]

[0132] As in the above embodiment, the test coverage is calculated based on the number of test functions f and the total number of functions Fun. The test coverage M can be calculated using the following calculation model:

[0133]

[0134] As in the above embodiment, the average code execution time is calculated by collecting the total test time and lines of code accumulated by testers within the testing organization. The average code execution time N can be calculated using the following model:

[0135]

[0136] Where LOC is the code size, t i Let N be the testing time for the i-th tester. t The number of test participants is denoted as tanh(), which is the hyperbolic tangent function.

[0137] As in the above embodiment, the average test case duration is calculated by collecting the total cumulative test time of testers and the number of test cases T. The average test case duration O can be calculated using the following calculation model:

[0138]

[0139] The embodiments of this application will be described and illustrated below through preferred embodiments.

[0140] In this preferred embodiment, five testing organizations (A, B, C, D, and E) are interested in undertaking the task. The task-assigning party needs to quickly and economically select one testing organization to conduct the testing based on the application materials submitted by the five organizations, in order to ensure the testing progress and quality. The application materials include information about the testing organization and information about 10 testing items. After normalization processing, these are input into the comprehensive evaluation model, and the evaluation results for the five testing organizations are calculated as follows:

[0141] P A =0.8258, P B =0.7556,P C =0.9228, P D =0.8056, P E =0.7865

[0142] Therefore, the party issuing the task will give priority to assigning the test task to test organization C.

[0143] Based on the above steps, this application embodiment evaluates the testing organization from both the testing organization and testing project perspectives by establishing a basic evaluation model for the testing organization and a basic evaluation model for the testing project. It also clarifies the corresponding evaluation indicators and the measurement methods for the values ​​of the evaluation indicators. This approach can effectively, quickly, and economically conduct a comprehensive evaluation of the testing organization, providing important guidance for the task allocation decision-making of the task assignor. It is simple to implement, convenient, and fast. At the same time, when the evaluation indicators change, the model can be quickly updated to obtain a new evaluation model. It is easy to modify and has a wide range of applications.

[0144] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0145] Furthermore, this application provides a comprehensive evaluation system for software testing organizations based on AHP and BPNN, used to implement the comprehensive evaluation method for software testing organizations based on AHP and BPNN as described in the above embodiments. (Refer to...) Figure 5As shown, the system includes:

[0146] Evaluation index system construction module 1 is used to construct an evaluation index system for testing organizations and an evaluation index system for testing projects based on an evaluation index for testing organizations and an evaluation index for testing projects, respectively.

[0147] Module 2, which establishes a basic evaluation model for testing organizations, is used to establish a basic evaluation model for testing organizations based on the evaluation indicators of testing organizations using the analytic hierarchy process.

[0148] Module 3, which establishes a basic evaluation model for test items, is used to establish and train a basic evaluation model for test items based on the evaluation indicators of test items using the analytic hierarchy process and BPNN.

[0149] The comprehensive evaluation model building module 4 is used to combine the basic evaluation model of the testing organization and the average evaluation model of the pre-selected number of test items using the analytic hierarchy process to obtain the comprehensive evaluation model. When a portion of the test items are selected to evaluate the testing organization, the basic evaluation model of the test items is the average evaluation model of the test items.

[0150] Module 5, which acquires evaluation results, is used to collect evaluation indicators for testing organizations and testing items, input them into the comprehensive evaluation model, and calculate the evaluation result P. Z .

[0151] In some embodiments, the testing facility basic evaluation model establishment module 2 further includes:

[0152] The judgment matrix construction module 201 is used to construct the judgment matrix A based on the relative importance of the evaluation indicators of the testing organization. jud =(a ij ) n×n ;

[0153] Consistency check module 202 is used to calculate the largest eigenvalue λ of the judgment matrix. max After determining the corresponding eigenvector S, calculate the consistency ratio CR. If the consistency ratio CR < 0.1, then the matrix passes the consistency test.

[0154] The indicator weight vector acquisition module 203 is used to normalize the feature vector S to obtain the indicator weight vector ω.

[0155] The testing organization basic evaluation model determination module 204 is used to obtain the testing organization basic evaluation model by weighted summation of the testing organization evaluation indicators and the indicator weight vector ω.

[0156] In some embodiments, the test item basic evaluation model establishment module 3 further includes:

[0157] BP neural network model building module 301 is used to configure BP neural network parameters and build a three-layer BP neural network structure model;

[0158] The training sample construction module 302 is used to select a set of test items, evaluate the test items using the analytic hierarchy process based on the evaluation indicators of the test items, and use the input values ​​of the indicators of all test items and the corresponding evaluation results as training samples.

[0159] The network model training module 303 is used to input training samples into the BP neural network model for training and obtain the weights of each neuron, wherein the weight matrix from the input layer to the hidden layer is W, and the weight vector from the hidden layer to the output layer is V.

[0160] The test item basic evaluation model determination module 304 is used to determine the test item basic evaluation model based on the weight matrix W and the weight vector V.

[0161] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0162] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A comprehensive evaluation method for software testing organizations based on AHP and BPNN, characterized in that, include: The evaluation indicator system is constructed by building an evaluation indicator system for testing organizations and an evaluation indicator system for testing projects, based on a set of evaluation indicators for testing organizations and a set of evaluation indicators for testing projects. The evaluation indicator system for testing organizations includes: qualifications, personnel capabilities, testing quality, contract progress, price, service quality, ability to accept urgent tasks, testing tools, and external environment. The evaluation indicator system for testing projects includes: test type, test case coverage, requirement coverage, test coverage, average code execution time, and average test case execution time. The steps for establishing a basic evaluation model for testing organizations are as follows: Based on the evaluation indicators of testing organizations, a basic evaluation model for testing organizations is established using the analytic hierarchy process. The steps for establishing a basic evaluation model for test items are as follows: Based on the evaluation indicators of the test items, a basic evaluation model for test items is established and trained using the analytic hierarchy process and BPNN. The steps for establishing a comprehensive evaluation model are as follows: The comprehensive evaluation model is obtained by combining the basic evaluation model of the testing organization and the average evaluation model of a pre-selected number of test items using the analytic hierarchy process. When a portion of the test items are selected to evaluate the testing organization, the basic evaluation model of the test items is the average evaluation model of the test items. The evaluation result acquisition steps involve collecting the evaluation indicators of the testing organization and the evaluation indicators of the testing items, inputting them into the comprehensive evaluation model, and calculating the evaluation result. ; The steps for establishing a basic evaluation model for testing organizations further include: The judgment matrix construction step involves constructing a judgment matrix based on the relative importance of the evaluation indicators of the testing organization. The relative importance scale ranges from [1, 9]. ; The consistency check step involves calculating the largest eigenvalue of the judgment matrix. and the corresponding feature vectors Then, calculate the consistency ratio. If the consistency ratio If the condition is met, the judgment matrix passes the consistency check; otherwise, return to the judgment matrix construction step for correction. The steps to obtain the indicator weight vector are as follows: [The text abruptly ends here, likely due to an incomplete sentence or Normalization yields the index weight vector ; The steps for determining the basic evaluation model for testing organizations include: matching the testing organization evaluation indicators with the indicator weight vector. The basic evaluation model of the testing organization is obtained by performing a weighted summation. The steps for establishing the basic evaluation model for the test item further include: Steps for building a BP neural network model: Configure BP neural network parameters and build a three-layer BP neural network structure model; The training sample construction steps are as follows: Select a set of test items, evaluate the test items using the analytic hierarchy process based on the evaluation indicators of the test items, and use the input values ​​of the indicators of all test items and the corresponding evaluation results as training samples. The network model training steps involve inputting the training samples into the BP neural network model for training, obtaining the weights of each neuron, where the weight matrix from the input layer to the hidden layer is... The weight vector from the hidden layer to the output layer is ; The steps for determining the basic evaluation model for the test project are based on the weight matrix. and weight vector The basic evaluation model for the test item is determined, and the basic evaluation model for the test item can be calculated using the following calculation model: ， In the formula, To test the basic evaluation results of the project, The number of neurons in the input layer. This represents the number of neurons in the hidden layer. For the input layer The actual score of each evaluation indicator For the input layer The indicators are transmitted to the hidden layer. The weights of each neuron, For the hidden layer The weights from each neuron to the output layer neurons. This is the neuron activation function.

2. The comprehensive evaluation method for software testing organizations based on AHP and BPNN according to claim 1, characterized in that, The comprehensive evaluation model is represented by the following calculation model: ， in, The evaluation results are based on the basic evaluation model of the testing organization. for The weight, The evaluation result of the average evaluation model for the test items. for The weight.

3. The comprehensive evaluation method for software testing organizations based on AHP and BPNN according to claim 2, characterized in that, Evaluation results of the average evaluation model for the test items The following calculation model was used to obtain the following: ， Where N is the number of selected test items, The evaluation results are for the basic evaluation model of the test item.

4. The comprehensive evaluation method for software testing organizations based on AHP and BPNN according to claim 1, characterized in that, The consistency ratio is calculated based on the following calculation model: ， Among them, consistency index Average random consistency index By looking up the table, it was found that the table has pre-configured [something]. The corresponding RI value.

5. A comprehensive evaluation system for software testing organizations based on AHP and BPNN, used to implement the comprehensive evaluation method for software testing organizations based on AHP and BPNN as described in any one of claims 1-4, characterized in that, include: The evaluation index system construction module is used to construct an evaluation index system for testing organizations and an evaluation index system for testing projects based on an evaluation index for a testing organization and an evaluation index for a testing project, respectively. The module for establishing a basic evaluation model for testing organizations is used to establish a basic evaluation model for testing organizations based on the evaluation indicators of the testing organizations using the analytic hierarchy process. The test project basic evaluation model establishment module is used to establish and train a test project basic evaluation model based on the test project evaluation indicators using the analytic hierarchy process and BPNN. The comprehensive evaluation model building module is used to combine the basic evaluation model of the testing organization and the average evaluation model of a pre-selected number of test items using the analytic hierarchy process to obtain a comprehensive evaluation model. When a portion of the test items are selected to evaluate the testing organization, the basic evaluation model of the test items is the average evaluation model of the test items. The evaluation result acquisition module is used to collect the evaluation indicators of the testing organization and the evaluation indicators of the testing items, input them into the comprehensive evaluation model, and calculate the evaluation results. .

6. The comprehensive evaluation system for software testing organizations based on AHP and BPNN according to claim 5, characterized in that, The module for establishing a basic evaluation model for testing organizations further includes: The judgment matrix construction module is used to construct a judgment matrix based on the relative importance of the evaluation indicators of the testing organization. ; The consistency verification module is used to calculate the largest eigenvalue of the judgment matrix. and the corresponding feature vectors Then, calculate the consistency ratio. If the consistency ratio If so, the judgment matrix passes the consistency test; The indicator weight vector acquisition module is used to obtain feature vectors. Normalization yields the index weight vector ; The testing organization basic evaluation model determination module is used to determine the testing organization evaluation indicators and the indicator weight vector. The basic evaluation model of the testing organization is obtained by performing a weighted summation.

7. The comprehensive evaluation system for software testing organizations based on AHP and BPNN according to claim 5, characterized in that, The basic evaluation model establishment module for the test item further includes: The BP neural network model building module is used to configure BP neural network parameters and build a three-layer BP neural network structure model. The training sample construction module is used to select a set of test items, evaluate the test items using the analytic hierarchy process based on the evaluation indicators of the test items, and use the input values ​​of the indicators of all test items and the corresponding evaluation results as training samples. The network model training module is used to input the training samples into the BP neural network model for training, and to obtain the weights of each neuron, wherein the weight matrix from the input layer to the hidden layer is as follows: The weight vector from the hidden layer to the output layer is ; The test project basic evaluation model determination module is used to determine the model based on the weight matrix. and weight vector Determine the basic evaluation model for the test items.

Images