Timely software quality guarantee method based on automatic defect detection and software testing
By combining the collaborative mechanism of automated defect detection and software testing, the problems of model drift and excessive resource consumption in agile development are solved, efficient and accurate defect detection and testing are achieved, and rapid iteration of agile development is supported.
Patent Information
- Application Number
- CN202510850482.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-24
- Publication Date
- 2025-10-03
AI Technical Summary
In the agile development and continuous integration/continuous delivery model, existing technologies have problems such as model performance being affected by concept drift, defect detection delays, and excessive consumption of traditional testing resources, making automated quality assurance difficult to be compatible with the agile development model.
By building a collaborative assurance mechanism for automated defect detection and software testing, combining the real-time risk prediction of machine learning models with traditional testing techniques, and using intelligent selectors to dynamically associate model prediction results with test execution strategies, we can generate automated test cases related to code changes and achieve an end-to-end quality assurance process without human intervention.
While ensuring the timeliness of detection, it improves the accuracy of defect verification, reduces invalid testing overhead, alleviates model performance degradation, achieves an iteration rate that matches the agile development model, and eliminates the efficiency bottleneck of manual verification.
Smart Images
Figure CN120743324A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a timely software quality assurance method based on automated defect detection and software testing, and belongs to the technical field of software quality assurance. Background Art
[0002] In modern software development practices dominated by agile development and continuous integration / continuous delivery (CI / CD) models, automated quality assurance technology has become a core means of ensuring software reliability. Currently, mainstream automated quality assurance solutions mainly include two technical paths:
[0003] 1. Real-time defect detection technology based on machine learning
[0004] Timely defect detection technology uses predictive models to conduct real-time risk assessments on code changes, offering the advantages of zero manual intervention and millisecond-level response. However, this technology has the following inherent drawbacks:
[0005] 1. Model performance is significantly constrained by concept drift, where changes in code distribution characteristics during software evolution can lead to a decline in prediction accuracy.
[0006] 2. To mitigate concept drift, an incremental learning strategy is required. However, obtaining defect labels for new versions of code is subject to verification delays. Developers must undergo lengthy manual reviews or wait for defects to be exposed before labeling can be completed, resulting in delayed model updates.
[0007] 2. Traditional Software Testing Technology
[0008] Although traditional software testing technology can provide deterministic defect verification results through test case execution, test case generation requires a large amount of computing resources. Even if historical test suites are reused, the execution time of the full set of test cases is still orders of magnitude different from the frequency of code submission. If the complete testing process is triggered for each code change, it will directly lead to an extension of the iteration cycle and fail to meet the delivery rate requirements of agile development.
[0009] To address this contradiction, when a defect detection model outputs a risk warning, existing technical solutions attempt to obtain more timely training data by having testers prioritize verifying the defect labels of the latest code changes. However, this solution introduces additional labor costs, which defeats the original purpose of automated quality assurance.
[0010] What is more noteworthy is that there are two major technical gaps in existing research:
[0011] First, there is no automated test case generation mechanism for code change scenarios. Existing work assumes that a complete test set already exists in the test repository, but does not consider the common problem of missing test cases in industrial scenarios.
[0012] Second, there is no mechanism to link change impact analysis with test case selection and execution. Blindly executing all test cases or random sampling tests will lead to invalid testing overhead.
[0013] In a real industrial environment with frequent functional iterations and limited testing resources, the above defects lead to a significant gap between academic achievements and engineering practice. Summary of the Invention
[0014] In order to solve the problems existing in the background technology, the present invention provides a timely software quality assurance method based on automated defect detection and software testing.
[0015] To achieve the above objectives, the present invention adopts the following technical solution: a timely software quality assurance method based on automated defect detection and software testing, the method comprising the following steps:
[0016] S1: Pre-training a timely defect prediction model based on annotated previous code changes , used to predict the probability that a new code change is defective;
[0017] S2: Whenever there is a new code change When generating, use pre-trained real-time defect prediction models Predict the probability that the current code change is defective ,and ;
[0018] S3: Based on the output of the timely defect prediction model and historical information, a selector is used to determine whether the current code change requires testing.
[0019] S4: Check if there is a test suite related to the current code change in the code base;
[0020] S5: If a test suite exists, select test cases that are relevant to the current code change and are not outdated and stable; if no test suite exists or the selected test cases are empty, use an automated test case generation algorithm to generate test cases relevant to the current code change;
[0021] S6: If S4 cannot find an existing test suite or the test case selected by S5 is empty, then use the automated test case generation algorithm to generate test cases related to the current code change;
[0022] S7: The framework will execute the test cases selected in S5 or the test cases generated in S6 to determine whether the current code changes introduce defects.
[0023] S8: If defects are found when executing the test case, the timely defect detection model is updated using the current code changes.
[0024] Furthermore, the judgment rules of the selector in S3 include:
[0025] Selector sets fixed threshold , when the prediction probability of the defect prediction model is When the selector determines that the current code change needs to be tested, otherwise, the selector determines that the current code change does not need to be tested and ends;
[0026] or:
[0027] Selector sets dynamically adjusted thresholds , which changes dynamically based on whether the past prediction results of the timely defect prediction model are the same as the actual results. When the prediction results are the same as the actual results, the threshold will increase, otherwise it will decrease. When the prediction probability of the current timely defect prediction model is When the selector determines that the current code change needs to be tested, otherwise, the selector determines that the current code change does not need to be tested and ends.
[0028] Furthermore, the S4 includes the following steps:
[0029] S401: Obtain the relative path of the function file of the code change, and filter the test code file through file type matching and keyword matching;
[0030] S402: Read the configuration file and determine the preset test warehouse path. If the test warehouse path exists in the configuration file, proceed to S404; otherwise, proceed to S403;
[0031] S403: traverse the current code repository to obtain the test repository path; and perform a test repository check. If a test repository exists, proceed to S404; otherwise, it indicates that the query is successful and returns null;
[0032] S404: Change the relative path of the function file and the corresponding test file; and perform a corresponding test file determination. If a corresponding test file exists, it indicates that the query is successful, and the relevant test suite is returned to inform the framework for further screening; otherwise, it indicates that the query fails, and returns empty, informing the framework that there is no corresponding test case.
[0033] Furthermore, the S5 includes the following steps:
[0034] S501: Select test cases related to the current code change;
[0035] S502: If the test case set selected in S501 is not empty, then the test case is judged to be outdated; otherwise, it returns null;
[0036] S503: If the test case set after filtering in S502 is not empty, then perform test case stability judgment; otherwise, return null;
[0037] S504: If the test case set after filtering in S503 is not empty, the filtered test cases are returned.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] The present invention breaks through the limitations of a single technical path by constructing a collaborative guarantee mechanism for automated defect detection and software testing, integrating the real-time risk prediction capability of machine learning models with the deterministic verification advantages of traditional testing technologies, thereby improving the accuracy of defect verification while ensuring the timeliness of detection; a dynamic association mechanism between model prediction results and test execution strategies is established through an intelligent selector to avoid redundant testing of low-risk code changes, significantly reducing the time overhead brought by full testing; the defect detection model is updated with real-time feedback of real defect labels of test results, forming a data closed loop to alleviate the problem of model performance degradation caused by concept drift; an automated test case generation method for code change scenarios is proposed, and a precise mapping mechanism between change impact domains and test case sets is established, effectively solving the industry pain points of missing test cases and invalid execution in industrial scenarios; finally, through the design of an end-to-end quality assurance process without human intervention, an iteration rate matching the agile development model is achieved while ensuring software reliability, eliminating the efficiency bottleneck introduced by the manual verification link in traditional solutions. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 is a flow chart of the present invention;
[0041] Figure 2 is the flow chart of S4;
[0042] Figure 3 This is a flowchart of S5. DETAILED DESCRIPTION
[0043] The technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0044] A timely software quality assurance method based on automated defect detection and software testing, the method comprising the following steps:
[0045] S1: Engineers pre-train a timely defect prediction model based on annotated previous code changes , used to predict the probability that a new code change is defective;
[0046] Pre-trained real-time defect prediction models can effectively predict past code changes, but recent code changes may have concept drift, leading to prediction errors. In practice, you can choose any machine learning model based on the situation.
[0047] S2: Whenever there is a new code change When generating, use pre-trained real-time defect prediction models Predict the probability that the current code change is defective ,and The timely defect prediction model is a binary classification model that takes as input the feature representation of the code change and outputs whether there is a defect. If there is a defect, the output is "1", otherwise the output is "0".
[0048] S3: Based on the output of the timely defect prediction model and historical information, a selector is used to determine whether the current code change requires testing.
[0049] S4: Check if there is a test suite related to the current code change in the code base;
[0050] Because generating a test suite is time-consuming, executing existing test cases can save a lot of time. The entire code repository includes the functional code repository and the test code repository, which together form a complex tree structure with leaf nodes being files.
[0051] S4 needs to find the test files corresponding to the changed code. A test file stores a set of test cases, also known as a test suite. Without any prior knowledge, traversing the entire tree to find the test files is time-consuming. Therefore, the framework incorporates prior knowledge to quickly find the test files corresponding to the changed code.
[0052] S5: If a test suite exists, further screen the test suite to select test cases that are relevant to the current code change, are not outdated, and are stable; if no test suite exists or the selected test cases are empty, use an automated test case generation algorithm to generate test cases relevant to the current code change;
[0053] S6: If S4 cannot find an existing test suite or the test case selected by S5 is empty, an automated test case generation algorithm is used to generate test cases related to the current code change. Common test case generation algorithms include test case generation based on genetic algorithms and test case generation using large language models.
[0054] S7: The framework will execute the test cases selected in S5 or the test cases generated in S6 to determine whether the current code changes introduce defects.
[0055] Testing can only prove the existence of defects in the software, but cannot prove the absence of defects in the software. If the test case does not find defects, it is impossible to determine whether the current code change is truly defect-free. Therefore, the current code change cannot be labeled immediately and cannot be used to update the timely defect detection model.
[0056] S8: If S7 discovers a defect by executing the test case, it means that the current code change can immediately obtain the "defective" label because the current code change is the latest sample. In this case, the current code change is used to update the timely defect detection model to alleviate the concept drift problem.
[0057] Furthermore, the judgment rules of the selector in S3 include:
[0058] Selector sets fixed threshold , when the prediction probability of the defect prediction model is When the selector determines that the current code change needs to be tested, otherwise, the selector determines that the current code change does not need to be tested and ends;
[0059] Using selectors to test only a subset of code changes, rather than all code changes, can significantly reduce testing time. Different selectors filter code changes to varying degrees, resulting in varying time savings and model performance. Greater time savings often means more buggy code changes are left untested.
[0060] At this time, the threshold does not change dynamically over time. A higher threshold means stricter screening conditions, fewer code changes will be tested, and the time saving rate is higher. However, fewer software tests mean that the framework will obtain less new data, and even the defect prediction model cannot fit the new data in time. Once concept drift occurs, the model performance will decline, further affecting the performance of the entire framework.
[0061] or:
[0062] More complex selectors often combine the prediction results of past and current defect prediction models to decide whether to test. Specifically, the selector sets a dynamically adjusted threshold. , which changes dynamically based on whether the past prediction results of the timely defect prediction model are the same as the actual results. When the prediction results are the same as the actual results, the threshold will increase, otherwise it will decrease; the selection of the initial threshold and the amplitude of the threshold change will affect the strictness of the selector screening. When the current prediction probability of the timely defect prediction model is When the selector determines that the current code change needs to be tested, otherwise, the selector determines that the current code change does not need to be tested and ends.
[0063] Furthermore, the S4 includes the following steps:
[0064] S401: Test files often correspond to functional files. Therefore, the first step is to obtain the relative path of the functional file with code changes and filter the test code files by file type matching and keyword matching.
[0065] Common version control systems, such as git, maintain change information for each code change. The change information stores the relative path of each changed file. Note that changed files often include function code-related files, test code-related files, configuration files, etc. The framework obtains the relative path of the changed function file through file type matching and keyword matching.
[0066] For example, in a Java project, the framework first determines whether the modified file is a Java file. This will retrieve all modified files related to functional and test code. Software development specifications require that the path to test files contain a "test" folder, and the test file name also contains the keyword "test." The framework then filters test code changes by keyword matching and ultimately retrieves the relative path to the modified functional file.
[0067] S402: Read the configuration file and determine the preset test warehouse path. If the test warehouse path exists in the configuration file, proceed to S404; otherwise, proceed to S403;
[0068] Test files are stored in the test repository. Searching for test files in the test repository significantly reduces the search space. Due to subtle differences in test repositories between projects, the framework cannot pre-set the relative path of the test repository. Developers must configure this by adding the repository path information to the configuration file provided by the framework.
[0069] S403: Traverse the current code repository to obtain the test repository path. Since the test repository path rarely changes within a project, the framework stores this path information in the configuration file for the next search. The framework then determines if a test repository exists. If so, the framework combines prior knowledge with the following: for example, for Java, a function file is: function code repository relative path / pathA / foo.java; a test file is: test code repository relative path / pathA / fooTest.java. Then, proceed to S404. Otherwise, the query is successful and returns nothing.
[0070] S404: The relative paths of the functional files and the corresponding test files need to comply with the above specifications or similar specifications. The framework uses a matching algorithm to obtain the relative paths of the test files and makes a corresponding test file determination. If a corresponding test file exists, it indicates that the query is successful, and the relevant test suite is returned to inform the framework for further screening. Otherwise, it indicates that the query has failed, and an empty string is returned to inform the framework that no corresponding test case exists.
[0071] Furthermore, the S5 includes the following steps:
[0072] S501: The framework integrates the most advanced test case selection algorithm to select test cases relevant to the current code changes;
[0073] S502: If the test case set selected in S501 is not empty, then the test case is judged as outdated. Outdated test cases refer to the situation where some test cases cannot be executed or the execution does not produce the expected results due to code changes. Executing outdated test cases will not help developers find defects, but may mislead them, and executing outdated test cases will waste time. The framework uses advanced outdated test case detection technology to filter out outdated test cases. Otherwise, it returns null.
[0074] S503: If the test case set filtered in S502 is not empty, the test case stability is determined. Some test cases may produce different results when executed multiple times. Such test cases are called unstable test cases. Unstable test cases, like outdated test cases, cannot find defects, mislead developers, and waste time. The framework uses advanced unstable test case detection technology to filter out unstable test cases. Otherwise, the result is empty.
[0075] S504: If the test case set after filtering in S503 is not empty, the filtered test cases are returned.
[0076] This invention combines a timely defect detection model with automated software testing, aiming to ensure the quality of every code change in modern, rapidly iterative software development, that is, to discover code defects introduced by code changes while significantly reducing time overhead and eliminating labor costs.
[0077] Generating and executing test cases, or simply executing existing test cases, is extremely time-consuming. Testing every code change significantly impacts software development speed. This invention eliminates the need to test every code change. Instead, it tests code changes that are more likely to be defective, as determined by a timely software defect detection model.
[0078] When software testing discovers a real defect in a code change, the code change is promptly and correctly labeled, and the defect detection model can be updated using the code change. This allows the model to adapt to the new data distribution more quickly, significantly reducing verification latency for the code change and alleviating concept drift.
[0079] After the timely software defect detection model outputs results, a selector is set to select the code changes that should be tested based on the model output and model history information, saving testing time while improving the performance of the entire framework. The selector is a key module that connects the timely defect prediction model and software testing. Different selector settings can result in significant differences in time savings and framework performance.
[0080] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.
[0081] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.
Claims
1. A timely software quality assurance method based on automated defect detection and software testing, characterized by: The method comprises the following steps: S1: Pre-training a timely defect prediction model based on annotated previous code changes , used to predict the probability that a new code change is defective; S2: Whenever there is a new code change When generating, use pre-trained real-time defect prediction models Predict the probability that the current code change is defective ,and ; S3: Based on the output of the timely defect prediction model and historical information, a selector is used to determine whether the current code change requires testing. S4: Check if there is a test suite related to the current code change in the code base; S5: If a test suite exists, select test cases that are relevant to the current code changes, are not outdated, and are stable; If no test suite exists or the selected test case is empty, an automated test case generation algorithm is used to generate test cases related to the current code change; S6: If S4 cannot find an existing test suite or the test case selected by S5 is empty, then use the automated test case generation algorithm to generate test cases related to the current code change; S7: The framework will execute the test cases selected in S5 or generated in S6 to determine whether the current code changes have introduced defects. S8: If defects are found when executing the test case, the timely defect detection model is updated using the current code changes.
2. The method for timely software quality assurance based on automated defect detection and software testing according to claim 1, characterized in that: The judgment rules of the selector in S3 include: Selector sets fixed threshold , when the prediction probability of the defect prediction model is When the selector determines that the current code change needs to be tested, otherwise, the selector determines that the current code change does not need to be tested and ends; or: Selector sets dynamically adjusted thresholds , which changes dynamically based on whether the past prediction results of the timely defect prediction model are the same as the actual results. When the prediction results are the same as the actual results, the threshold will increase, otherwise it will decrease. When the prediction probability of the current timely defect prediction model is When the selector determines that the current code change needs to be tested, otherwise, the selector determines that the current code change does not need to be tested and ends.
3. The timely software quality assurance method based on automated defect detection and software testing according to claim 1, characterized in that: The S4 comprises the following steps: S401: Obtain the relative path of the function file of the code change, and filter the test code file through file type matching and keyword matching; S402: Read the configuration file and determine the preset test warehouse path. If the test warehouse path exists in the configuration file, proceed to S404; Otherwise, proceed to S403; S403: Traverse the current code repository to obtain the test repository path; Then, the test warehouse is determined. If a test warehouse exists, proceed to S404. Otherwise, it means that the query is successful and returns empty; S404: Change the relative path of the function file and the corresponding test file; and perform a corresponding test file determination. If a corresponding test file exists, it indicates that the query is successful, and the relevant test suite is returned to inform the framework for further screening; otherwise, it indicates that the query fails, and returns empty, informing the framework that there is no corresponding test case.
4. The method for timely software quality assurance based on automated defect detection and software testing according to claim 1, characterized in that: The S5 comprises the following steps: S501: Select test cases related to the current code change; S502: If the test case set selected in S501 is not empty, then the test case is judged to be outdated; otherwise, it returns null; S503: If the test case set after filtering in S502 is not empty, then perform test case stability judgment; Otherwise, it returns empty; S504: If the test case set after filtering in S503 is not empty, the filtered test cases are returned.