A method for detecting and verifying bias in software automatic repair evaluation

Abstract

The present invention discloses a method for discovering and verifying bias in software automatic repair evaluation, including: step S1: collecting relevant literature; step S2: screening literature; step S3: obtaining discovered bias: extracting discovered bias from the bias research literature obtained in step S2; step S4: constructing tree classification: proposing a tree classification for automatic repair technology evaluation bias; step S5: mapping existing bias: mapping the bias obtained in step S3 to the tree proposed in step S4; step S6: discovering new bias: discovering new bias based on the tree classification bias mapping obtained in step S5; step S7: implementing an evaluation software platform; step S8: eliminating existing bias: eliminating existing bias based on the software platform; step S9: verifying new bias: after eliminating existing bias, verifying new bias on the evaluation software platform. The present invention has the advantages of simple principle, easy operation, high verification accuracy, etc.

Description

Technical Field

[0001] The present invention mainly relates to the field of verification technology of software engineering, and in particular to a bias discovery and verification method for software automatic repair evaluation based on evidence-based software engineering and taxonomy. Background Art

[0002] As a research hotspot in the field of software engineering, software automatic repair technology aims to automatically repair defects in software without human intervention. Its purpose is to reduce the debugging burden of developers, improve software quality, and save development costs. It has great value and significance.

[0003] Usually, every time a new software automatic repair technology is proposed, an evaluation activity is required to accurately determine its advantages and disadvantages compared with previous repair technologies. The evaluation of traditional software automatic repair technology consists of three steps:

[0004] S1: Select an internationally accepted benchmark defective program dataset and select the software automatic repair technology for evaluation.

[0005] S2: For each defective program in the selected benchmark defective program data set, a repair experiment is performed using software automatic repair technology to obtain a repair result, that is, a repaired patch program and repair time.

[0006] S3: Analyze the repair results and compare the advantages and disadvantages of different software automatic repair technologies.

[0007] In the above steps, the evaluator often introduces various biases in the evaluation process. The bias is defined as any factor that affects the evaluation results in the automatic repair evaluation. Such bias often leads to inaccurate evaluation results. For example, in the data set selection step, the evaluator selects data set M for automatic repair technology A and data set N for automatic repair technology B, and finally compares the repair results of the two. In this case, the evaluator is considered to have introduced a data set selection bias, resulting in A and B being unable to be fairly evaluated, thereby causing distortion of the evaluation results.

[0008] Therefore, since the rise of software automatic repair technology, bias research, that is, discovering and verifying the bias in software automatic repair evaluation, has become an important topic. However, the current methods of discovering bias rely heavily on the researcher's personal subjective understanding of automatic repair evaluation, and there is currently no method that can support systematic bias discovery in software automatic repair evaluation.

[0009] In the bias research of software automatic repair evaluation, it is found that the evaluation bias often involves the verification of bias. Traditional bias verification usually consists of three steps:

[0010] S1: Multiple settings for factors corresponding to evaluation bias.

[0011] S2: Evaluate automatic repair techniques for biases in different settings through controlled variable experiments.

[0012] S3: Collect the evaluation results of automatic repair technology and evaluate the impact of this bias on the evaluation of automatic repair technology.

[0013] However, current bias assessment technology often introduces other biases when verifying one bias, resulting in distortion of the final bias verification results.

[0014] As can be seen from the above, the above traditional methods have the following problems:

[0015] 1. Low bias detection efficiency: Software automatic repair evaluation can be divided into three steps, namely: data set selection, repair experiment, and effect evaluation. Due to the complexity of the software automatic repair system, evaluators often introduce various biases when conducting software automatic repair evaluation. Existing bias detection technology relies too much on researchers' subjective understanding of automatic repair evaluation, and can often only detect one bias at a time. Currently, there is no technology that can support systematic bias detection.

[0016] 2. Poor accuracy of bias verification: In the process of bias verification, the control variable method needs to be adopted, that is, except for the bias to be verified, other factors that can affect the experimental results need to be kept consistent. However, the current bias verification technology often ignores the existence of other biases when verifying a specific bias, thereby introducing other biases into the bias evaluation experiment, which ultimately affects the accuracy of bias verification. Summary of the invention

[0017] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a method for discovering and verifying bias in automatic software repair evaluation, which has a simple principle, is easy to operate and has high verification accuracy.

[0018] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0019] A method for discovering and verifying bias in software automatic repair evaluation, comprising:

[0020] Step S1: Collect relevant literature: Collect literature related to the evaluation of software automatic repair technology;

[0021] Step S2: Screening literature: Screen the literature collected in step S1 to obtain literature on bias research;

[0022] Step S3: Obtaining discovered biases: extracting discovered biases from the bias research literature obtained in step S2;

[0023] Step S4: Constructing a tree classification: Proposing a tree classification for evaluating bias in automatic repair techniques;

[0024] Step S5: Mapping existing biases: Mapping the biases obtained in step S3 to the tree proposed in step S4;

[0025] Step S6: Discovering new biases: Based on the tree classification bias mapping obtained in step S5, discover new biases;

[0026] Step S7: Implementing an evaluation software platform: Implementing a software platform for evaluating bias;

[0027] Step S8: Eliminating existing bias: Eliminating existing bias based on the software platform implemented in step S7;

[0028] Step S9: Verify the new bias: After eliminating the existing bias, verify the new bias on the evaluation software platform.

[0029] As a further improvement of the method of the present invention: the step S1 comprises:

[0030] Step S11: constructing a search string based on AND and OR in Boolean operations;

[0031] Step S12: Search the character string constructed in step S11 in the literature database and academic literature search engine, and collect the literature corresponding to the search results.

[0032] As a further improvement of the method of the present invention: the step S2 comprises:

[0033] Step S21: Based on the relevant documents acquired in step S1, documents whose full texts are not available are excluded;

[0034] Step S22: Exclude the dissertation.

[0035] Step S23: Exclude the literature that has not conducted bias research.

[0036] As a further improvement of the method of the present invention: the step S3 comprises:

[0037] Step S31: Based on the literature on bias research obtained in S2, extract the bias discovered and verified in each bias research;

[0038] Step S32: extract basic information of each bias study, including: author name, channel of research publication, and year of publication;

[0039] Step S33: extracting the automatic repair technology evaluation index affected by the bias and the magnitude of the impact.

[0040] As a further improvement of the method of the present invention: the step S4 comprises:

[0041] Step S41: Determine that the classification type is tree classification;

[0042] Step S42: determining the classification criteria as a specific stage of the software automatic repair assessment affected by bias;

[0043] Step S43: Based on the principle of mutual independence and complete exhaustiveness, the biases are classified, and finally a three-layer tree classification is obtained.

[0044] As a further improvement of the method of the present invention: Step S5 comprises:

[0045] Step S51: Based on the discovered bias obtained in step S3, first map it to the second level classification of the classification obtained in S4;

[0046] Step S52: further map the discovered bias to the third level of classification obtained in S4, i.e., the lowest level of classification.

[0047] As a further improvement of the method of the present invention: Step S6 comprises:

[0048] Step S61: Based on the tree classification constructed in step S4 and the bias mapping found in step S5, find a classification in the third-level classification that has not yet been mapped;

[0049] Step S62: Based on the classification that has not yet been mapped, identify the corresponding bias that has not yet been discovered;

[0050] As a further improvement of the method of the present invention: the step S7 comprises:

[0051] Step S71: Implementing a data set selection module;

[0052] Step S72: Implement a defect location module, that is, implement a standard defect location technology Ochiai into the software platform;

[0053] Step S73: Implementing a patch generation module;

[0054] Step S74: Implementing a patch verification module;

[0055] Step S75: Implement an effect evaluation module to collect and analyze the results of the bias evaluation experiment, including the patch program and the consumed time.

[0056] As a further improvement of the method of the present invention: the step S8 comprises:

[0057] Step S81: among the biases found, identify the biases that can be eliminated;

[0058] Step S82: Eliminate bias, that is, use a unified configuration to control variables.

[0059] As a further improvement of the method of the present invention: the specific steps of step S9 are:

[0060] Step S91: conducting a control variable experiment, that is, implementing a variety of settings corresponding to the bias to be verified;

[0061] Step S92: conducting a software automatic repair technology evaluation experiment for each setting;

[0062] Step S93: Evaluate the experimental results to verify the impact of bias on the evaluation of software automatic repair technology.

[0063] Compared with the prior art, the advantages of the present invention are:

[0064] 1. The software automatic repair assessment bias discovery and verification method of the present invention is based on the systematic literature review method in evidence-based software engineering, systematically understands the existing biases that have been discovered, and then maps the existing biases one by one to the bias classification based on the tree classification method in taxonomy, so as to systematically discover new biases, with the aim of improving the current bias discovery efficiency.

[0065] 2. The software automatic repair assessment bias discovery and verification method of the present invention is based on the bias distribution discovered through a systematic literature review, clarifies the bias that can be eliminated, and then implements a bias assessment software platform based on a layered software architecture to eliminate the identified bias that can be eliminated and provide an experimental platform for bias verification, with the aim of improving the current accuracy of bias verification.

[0066] 3. The software automatic repair assessment bias discovery and verification method of the present invention improves the efficiency of bias discovery: Compared with the existing bias discovery technology that heavily relies on the subjective understanding of researchers, the present invention simultaneously utilizes the systematic literature review method based on evidence-based software engineering and the tree classification method based on taxonomy to systematically explore bias and improve the efficiency of bias discovery.

[0067] 4. The software automatic repair evaluation bias discovery and verification method of the present invention improves the accuracy of bias verification: Compared with the prior art that ignores the existence of other biases when verifying bias, thereby introducing bias into the bias evaluation experiment and affecting the accuracy of bias verification, the present invention constructs a bias distribution through a systematic literature review method, clarifies the bias that can be eliminated, and then implements a bias evaluation software platform based on a layered software architecture to eliminate the bias that can be eliminated, and then verifies the bias to be verified, thereby improving the accuracy of bias verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a schematic flow diagram of the method of the present invention.

[0069] Figure 2 It is a schematic diagram of the process of constructing tree classification in a specific application example of the present invention. DETAILED DESCRIPTION

[0070] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] like Figure 1 As shown, the method for discovering and verifying bias in automatic software repair evaluation of the present invention comprises the following steps:

[0072] S1: Collect relevant literature: Collect literature related to the evaluation of software automatic repair techniques.

[0073] S2: Screening literature: Screen the literature collected in S1 to obtain literature that conducts bias research.

[0074] S3: Obtain discovered biases: Extract discovered biases from the bias research literature obtained in S2.

[0075] S4: Constructing a tree-like classification: Propose a tree-like classification for evaluating bias in automatic repair techniques.

[0076] S5: Map existing biases: Map the biases obtained in S3 to the tree proposed in S4.

[0077] S6: Discover new biases: Discover new biases based on the tree classification bias mapping obtained in S5.

[0078] S7: Implement a software platform for assessing bias: Implement a software platform for assessing bias.

[0079] S8: Eliminate existing biases: Eliminate existing biases based on the software platform implemented in S7.

[0080] S9: Validate new biases: After eliminating existing biases, validate new biases on the evaluation software platform.

[0081] In a specific application example, the specific steps of step S1 are:

[0082] S11: Construct a search string based on AND and OR in Boolean operations: (automated OR automatic) AND (program OR software OR fault OR bug OR defect) AND (repair OR fix).

[0083] S12: Search the string constructed in S11 in five literature databases, including ACM digital library, IEEE Xplore, Springer Link, Elsevier Science Direct, and Wiley, as well as the academic literature search engine Google Scholar, and collect the literature corresponding to the search results.

[0084] In a specific application example, the specific steps of step S2 are:

[0085] S21: Based on the relevant literature obtained in S1, the literature whose full text is not available is excluded.

[0086] S22: Exclude theses and dissertations.

[0087] S23: Literature that did not conduct bias research was excluded.

[0088] In a specific application example, the specific steps of step S3 are:

[0089] S31: Based on the literature on bias research obtained in S2, extract the biases discovered and verified in each bias study.

[0090] S32: Extract basic information of each bias study, including: author name, channel where the study was published, and year of publication.

[0091] S33: Extract the automatic repair technology evaluation indicators affected by the bias and the magnitude of the impact.

[0092] In a specific application example, the specific steps of step S4 are:

[0093] S41: Determine that the classification type is tree classification.

[0094] S42: Determine the specific stages of the evaluation of automated software repair for the impact of bias by classifying criteria.

[0095] S43: Based on the principle of mutual independence and complete exhaustiveness, biases are classified and finally a three-layer tree classification is obtained, such as Figure 2 shown.

[0096] In a specific application example, the specific steps of step S5 are:

[0097] S51: Based on the discovered bias obtained in S3, first map it to the second level classification of the classification obtained in S4.

[0098] S52: The discovered bias is further mapped to the third level of classification obtained in S4, i.e., the lowest level of classification.

[0099] In a specific application example, the specific steps of step S6 are:

[0100] S61: Based on the tree classification constructed in S4 and the bias mapping found in S5, it is found that there is no mapping in the third-level classification.

[0101] S62: Based on the classification that has not yet been mapped, identify the corresponding undiscovered biases.

[0102] In a specific application example, the specific steps of step S7 are:

[0103] S71: Implement the data set selection module, that is, implement two internationally used defect program data sets, Defects4J and QuixBugs, into the software platform.

[0104] S72: Implement a defect location module, that is, implement a standard defect location technology Ochiai into the software platform.

[0105] S73: Implement the patch generation module, that is, implement four patch generation algorithms, including TBar, SimFix, Nopol and DynaMoth, into the software platform.

[0106] S74: Implement the patch verification module, that is, implement three patch verification strategies, including the naive strategy, the test case priority strategy, and the test case selection strategy, into the software platform.

[0107] S75: Implement the effect evaluation module to collect and analyze the results of the bias evaluation experiment, including the patch program and the consumed time.

[0108] In a specific application example, the specific steps of step S8 are:

[0109] S81: Identify the biases that have been discovered and which can be eliminated.

[0110] S82: Eliminate bias by using a unified configuration to control variables.

[0111] In a specific application example, the specific steps of step S9 are:

[0112] S91: Conduct controlled variable experiments, that is, implement multiple settings corresponding to the bias to be verified.

[0113] S92: Conduct experiments to evaluate software automatic repair techniques for each setting.

[0114] S93: Evaluate the experimental results to verify the impact of bias on the evaluation of software automatic repair technology.

[0115] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for discovering and verifying bias in automatic software repair evaluation, characterized in that: include: Step S1: Collect literature related to the evaluation of software automatic repair technology; Step S2: Screening literature: Screen the literature collected in step S1 to obtain literature on bias research; Step S3: Obtaining discovered biases: extracting discovered biases from the bias research literature obtained in step S2; Step S4: Constructing a tree classification: Proposing a tree classification for evaluating bias in automatic repair techniques; Step S5: Mapping existing biases: Mapping the biases obtained in step S3 to the tree proposed in step S4; Step S6: Discovering new biases: Based on the tree classification bias mapping obtained in step S5, discover new biases; Step S7: Implementing a software platform for assessing bias; Step S8: Eliminating existing bias: Eliminating existing bias based on the software platform implemented in step S7; Step S9: Verify the new bias: After eliminating the existing bias, verify the new bias on the evaluation software platform.

2. The method for discovering and verifying bias in automatic software repair evaluation according to claim 1, characterized in that: The step S1 comprises: Step S11: constructing a search string based on AND and OR in Boolean operations; Step S12: Search the character string constructed in step S11 in the literature database and academic literature search engine, and collect the literature corresponding to the search results.

3. The method for discovering and verifying bias in automatic software repair evaluation according to claim 2, characterized in that: The step S2 comprises: Step S21: Based on the relevant documents acquired in step S1, documents whose full texts are not available are excluded; Step S22: excluding thesis; Step S23: Exclude the literature that has not conducted bias research.

4. The method for discovering and verifying bias in automatic software repair evaluation according to claim 3, characterized in that: The step S3 comprises: Step S31: Based on the literature on bias research obtained in S2, extract the bias discovered and verified in each bias research; Step S32: extract basic information of each bias study, including: author name, channel of research publication, and year of publication; Step S33: extracting the automatic repair technology evaluation index affected by the bias and the magnitude of the impact.

5. The method for discovering and verifying bias in automatic software repair evaluation according to any one of claims 1 to 4, characterized in that: The step S4 comprises: Step S41: Determine that the classification type is tree classification; Step S42: determining the classification criteria as a specific stage of the software automatic repair assessment affected by bias; Step S43: Based on the principle of mutual independence and complete exhaustiveness, the biases are classified, and finally a three-layer tree classification is obtained.

6. The method for discovering and verifying bias in automatic software repair evaluation according to claim 5, characterized in that: The step S5 comprises: Step S51: Based on the discovered bias obtained in step S3, first map it to the second level classification of the classification obtained in S4; Step S52: further map the discovered bias to the third level of classification obtained in S4, i.e., the lowest level of classification.

7. The method for discovering and verifying bias in automatic software repair evaluation according to claim 6, characterized in that: The step S6 comprises: Step S61: Based on the tree classification constructed in step S4 and the bias mapping found in step S5, find a classification in the third-level classification that has not yet been mapped; Step S62: Based on the classification that has not yet been mapped, the corresponding bias that has not yet been discovered is identified.

8. The method for discovering and verifying bias in automatic software repair evaluation according to any one of claims 1 to 4, characterized in that: The step S7 comprises: Step S71: Implementing a data set selection module; Step S72: Implement a defect location module, that is, implement a standard defect location technology Ochiai into the software platform; Step S73: Implementing a patch generation module; Step S74: Implementing a patch verification module; Step S75: Implement an effect evaluation module to collect and analyze the results of the bias evaluation experiment, including the patch program and the consumed time.

9. The method for discovering and verifying bias in automatic software repair evaluation according to any one of claims 1 to 4, characterized in that: The step S8 comprises: Step S81: among the biases found, identify the biases that can be eliminated; Step S82: Eliminate bias, that is, use a unified configuration to control variables.

10. The method for discovering and verifying bias in automatic software repair evaluation according to any one of claims 1 to 4, characterized in that: The specific steps of step S9 are: Step S91: conducting a control variable experiment, that is, implementing a variety of settings corresponding to the bias to be verified; Step S92: conducting a software automatic repair technology evaluation experiment for each setting; Step S93: Evaluate the experimental results to verify the impact of bias on the evaluation of software automatic repair technology.

Images