Regression test result difference root cause positioning system and electronic equipment
By automatically analyzing the differences in regression test results using a large language model and locating code changes, this technology solves the problem of reliance on human experience in existing technologies and achieves efficient and low-cost investigation of differences in regression test results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-04-03
AI Technical Summary
In existing technologies, the investigation of differences in regression test results relies on human experience and specific standards, resulting in high rule maintenance costs, limited accuracy, and difficulty in efficiently improving regression testing efficiency.
By using a large language model combined with a target knowledge base, the system automatically analyzes the actual test results and the online version test results, locates the code change locations, and outputs the target code change locations that do not meet the test objectives and their corresponding target result difference fields.
It significantly improves the efficiency of regression test result analysis, reduces reliance on manual experience and system maintenance costs, and enhances positioning accuracy and efficiency.
Smart Images
Figure CN121785924A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a system and electronic device for locating the root causes of differences in regression test results. Background Technology
[0002] In API automation testing scenarios, differential testing regression is generally used. The same batch of test traffic is used to send requests to the online stable environment and the project code to be verified simultaneously. The return results of the two requests are compared to identify functional failures that may be introduced after code changes.
[0003] In related technologies, testers determine whether code changes might lead to changes in interface responses by modifying the code syntax tree. For example, modifying validation classes or return class attributes are considered high-risk changes. However, this approach not only heavily relies on human experience for rule formulation but also depends on specific standards for rule implementation. For instance, judging validation classes by class name keywords requires maintaining a list of keywords for validators, checkers, and filters, and relies on developers adhering to standards. This limits accuracy, increases maintenance costs, and requires continuous adjustments as the code under test is upgraded.
[0004] Therefore, it is necessary to reduce the cost of investigation and maintenance and improve the efficiency of regression testing.
[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0006] The purpose of this disclosure is to provide a regression test result root cause localization system and electronic device to reduce the cost of regression test result investigation and maintenance, and improve regression test efficiency.
[0007] According to a first aspect of the present disclosure, a regression test result difference root cause localization system is provided. The system includes: a test result acquisition module connected to a test platform, used to acquire actual test results and online version test results; an information acquisition module connected to a development platform, used to call the code change interface of the development platform to acquire code change information and test target information associated with the actual test results; and a large language model connected to a target knowledge base, used to automatically determine result difference fields based on the actual test results and the online version test results, determine the code change location corresponding to the result difference fields and whether the code change location conforms to the test target information based on the test target information and the code change information, and output the target code change location that does not conform to the test target information and the target result difference field corresponding to the target code change location.
[0008] According to a second aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to operate the system as described above based on instructions stored in the memory.
[0009] The system of this embodiment obtains code change information, test target information, and result difference fields, and uses a large language model to analyze and locate the code change location that causes the inconsistent return of the result difference field. It then judges whether it meets expectations and automatically outputs the target code change location that does not meet expectations and the target result difference field corresponding to the target code change location. This can significantly improve the efficiency of regression test result analysis.
[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0012] Figure 1 This is a flowchart of the root cause localization system for differences in regression test results in an exemplary embodiment of this disclosure.
[0013] Figure 2 This is a flowchart illustrating how a large language model determines the location of code changes in an exemplary embodiment of this disclosure.
[0014] Figure 3 This is a schematic diagram of the execution logic for root cause localization of differences in regression test results in an exemplary embodiment of this disclosure.
[0015] Figure 4 This is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation
[0016] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.
[0017] Furthermore, the accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0018] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0019] Figure 1 This is a flowchart of the root cause localization system for differences in regression test results in an exemplary embodiment of this disclosure.
[0020] refer to Figure 1 In this embodiment of the disclosure, the regression test result difference root cause localization system 100 includes: Test result acquisition module 1 connects to test platform 01 and is used to acquire actual test results and online version test results; Code information acquisition module 2 connects to development platform 02 and is used to call the code change interface of the development platform to obtain code change information and test target information associated with the test results. The large language model 3 connects to the target knowledge base 03 and is used to automatically determine the result difference fields based on the actual test results and the online version test results. It determines the code change location corresponding to the result difference fields and whether the code change location conforms to the test target information based on the test target information and code change information. It outputs the target code change location that does not conform to the test target information and the target result difference fields corresponding to the target code change location.
[0021] The system in this embodiment introduces a Large Language Model (LLM) to understand the type and content of interface changes, changed code, and experience and knowledge in plug-in troubleshooting. This enhances the model's understanding and ultimately locates the core changes causing inconsistent interface returns in regression test results. It also makes a preliminary judgment on whether the changes meet expectations, saving troubleshooting time for R&D and testing. Furthermore, it can disclose the reasoning behind the inference, enhancing the interpretability of the inference results.
[0022] By relying on a large model to understand code and changes, there is no strong dependence on human experience. Furthermore, by relying on understanding the code implementation rather than simply judging by class names, the accuracy of location can be significantly improved. Third, code replacement or upgrades do not require upgrading the large model, reducing maintenance costs. Finally, human experience and knowledge are only used as supplementary tools, and only textual descriptions of experience are needed; there is no need to create and maintain code for checking rules, which can greatly reduce system maintenance and upgrade costs.
[0023] In the exemplary embodiment, the actual test results include the interface call results, and the online version test results include the online interface call results. The interface call results refer to the complete feedback data obtained after initiating a call to the target system interface during the test process, including the response status code, response data, response time, and exception information. This data directly reflects the interface's operational status and functional implementation in the current test environment and is the core basis for determining whether the interface function is normal.
[0024] In this embodiment of the disclosure, the test result acquisition module 1 can establish a data transmission channel with the test platform 01 through a preset interface call protocol (such as HTTP, HTTPS, RPC, etc.), actively pull the test result data stored in the test platform 01 at preset time intervals, and can also receive the actual test results and online version test results actively pushed by the test platform 01 after the test task is completed. The module performs format verification, deduplication and preliminary sorting on the acquired test result data to ensure the integrity and validity of the data.
[0025] The testing platform 01 can be a dedicated system for executing regression testing tasks, with built-in core functions such as test case management, test environment deployment, test task scheduling, and test result storage. It can simulate real-world business scenarios to initiate test requests, record various data during the testing process, and initially correlate actual test results with pre-stored online version test results, providing a standardized source of test data for the test result acquisition module 1. In this embodiment, an existing testing platform can be modified to interface with system 100, responding to test result retrieval messages from the test result acquisition module 1 and returning actual test results and online version test results, or proactively pushing actual test results and online version test results after the test task is completed.
[0026] In some embodiments, both the test platform 01 and the test result acquisition module 1 directly provide the actual test results and the online version test results, thereby enabling the large language model 3 to acquire the actual test results and the online version test results, and automatically determine the result difference field (diff) corresponding to this regression test based on the actual test results and the online version test results.
[0027] In other embodiments, the test platform 01 or the test result acquisition module 1 can also directly compare the actual test results with the online version test results, output the result difference field corresponding to this regression test, and only transmit the result difference field to the large language model 3.
[0028] The code information acquisition module 2 can use the standardized API interface exposed by the development platform 02 to call the code change interface with key parameters such as test task identifier and test time range. It can then extract code change information related to the test task within the corresponding time period, such as code submission records, code modification content, modifier, and modification time, as well as test target information such as test scope, functional requirements, and performance indicators corresponding to the test task from the code version management module of the development platform 02.
[0029] In some embodiments, the information acquisition module 2 is further configured to call the configuration change log of the development platform 02 to obtain configuration change information associated with the actual test results. For example, the information acquisition module 2 can call the configuration change log query interface of the development platform 02 to obtain the configuration item modification records of the target system during the test, including configuration item name, value before modification, value after modification, reason for modification, modification time, and other configuration change information, thereby achieving comprehensive collection of code and configuration-level change information.
[0030] When the information acquisition module 2 is also used to call configuration change information, the large language model 3 determines the code change location corresponding to the result difference field and whether the code change location conforms to the test target information based on the test target information, code change information, and configuration change information. It then outputs the target code change location that does not conform to the test target information and the target result difference field corresponding to the target code change location.
[0031] Development platform 02, for example, is a development and operations platform that integrates code version control (such as GitLab, Gitee), project management, configuration management, and interface management. It can standardize the entire project development process, record all key information during code development and configuration modification, and provide a reliable information query and access point for code information acquisition module 2. In this embodiment, a conventional development platform can be modified to respond to data retrieval commands from code information acquisition module 2 and return data.
[0032] Large Language Model 3 can be a dedicated large model trained specifically for regression test analysis. Training data includes massive amounts of historical test cases from regression testing scenarios, difference location records, correlation data between code changes and result difference fields, configuration modification impact analysis reports, and professional knowledge such as code syntax rules, interface testing specifications, and troubleshooting logic. Through targeted training, this model possesses the ability to accurately identify the types of test result difference fields, quickly correlate code and configuration change nodes, and efficiently analyze the root causes of differences.
[0033] Test result acquisition module 1 and code information acquisition module 2 are respectively connected to the large language model 3. They actively transmit data to the large language model 3, or respond to the data call request of the large language model 3 and call the corresponding data from the test platform 01 and the development platform 02 respectively according to the test identification information provided by the large language model 3 and transmit it to the large language model 3.
[0034] In some embodiments, the large language model 3 can initiate the root cause localization task for regression test result discrepancies by manually inputting test identification information. For example, after manually inputting test identification information into the large language model 3, the large language model 3 provides the test identification information to the test result acquisition module 1 and the code information acquisition module 2 to obtain data. The data includes the actual test results and online version test results corresponding to the test identification information, as well as the code change information, test target information, and configuration change information associated with the actual test results. After successful data acquisition, the large language model 3 automatically starts analyzing the data and outputs the target code change locations that do not conform to the test target information and the target result difference fields corresponding to the target code change locations.
[0035] Among them, test identification information refers to the instructional code or script used to trigger regression test execution and clarify the test scope and objectives. It may include test case identifiers, interface call parameters, test environment configurations, expected result verification rules, data filtering conditions, etc. Its function is to provide the data collection location basis for test result acquisition module 1 and code information acquisition module 2. Test result acquisition module 1 can retrieve the actual test results (such as interface response data, execution status, exception logs, etc.) of the corresponding test task from test platform 01 based on the interface parameters, test case identifiers, etc. in the test identification information, as well as the matching online version test results (such as interface return data from historical stable environments). Code information acquisition module 1 can retrieve the corresponding code change records (such as modified file paths, code submission content, modifiers and times), test target information (such as functional requirements, performance indicators, compliance requirements, etc. of this test) and configuration change logs (such as system parameter adjustments, configuration item additions and subtractions during the test cycle) from development platform 02 based on the project identifier, test time range, functional module path, etc. associated with the test identification information.
[0036] In some embodiments, the large language model 3 can utilize the target knowledge base 03 to perform RAG (Retrieval-Augmented Generation) to perform a regression test result difference root cause localization task, for example, to perform code change location localization in the regression test result difference root cause localization task.
[0037] Figure 2 This is a flowchart illustrating how a large language model determines the location of code changes in an exemplary embodiment of this disclosure.
[0038] refer to Figure 2 In an exemplary embodiment, the process by which the large language model determines the location of code changes may include: Step S21: Determine the search keywords based on the result difference field, and search the target knowledge base based on the search keywords to determine the reasons for the differences; Step S22: Determine the code change segment based on the code change information, and verify the code change segment according to the candidate reasons for difference to determine whether the code change segment meets the candidate reasons for difference. Step S23: When the code change segment corresponds to the candidate difference reason, locate the code change position in the code change segment according to the candidate difference reason.
[0039] The target knowledge base 03 can include built-in experience in locating the root causes of result difference fields (provided manually) or other external knowledge. This knowledge information includes, but is not limited to, historical investigation experience documents, requirement specifications, detailed design documents, code comment libraries, etc., to provide various types of code changes that may lead to result difference fields or the mapping relationship between code changes and result difference fields, root cause cases of differences verified in historical regression tests, knowledge base of the impact of changes in programming language syntax rules and code logic, compatibility difference feature library corresponding to framework / component version changes, and data flow / control flow anomaly pattern library caused by code changes, etc.
[0040] In step S21, when determining the search keywords, the large language model 3 can determine the possible reasons for the differences corresponding to each result difference field (i.e., the diff field), i.e., the candidate reasons for the differences, based on the built-in knowledge of the target knowledge base 03. In the exemplary embodiment, the search keywords include any one or a combination of the name, type, and interface path to which the result difference field belongs.
[0041] For example, when the result difference field is a missing data field returned by the interface, the large language model 3 extracts search keywords such as interface return, missing field, data structure, and response construction based on the built-in knowledge of the mapping relationship between missing interface fields and code changes in the target knowledge base 03; when the result difference field is a deviation in numerical calculation results, it can combine the numerical calculation anomaly root cause case library in the target knowledge base 03 to determine search keywords such as numerical type conversion, operation logic modification, parameter passing error, and precision control.
[0042] Next, based on the aforementioned search keywords, a full-text search or structured query is performed on the target knowledge base 03. The large language model 3 then filters out candidate content with a matching degree higher than a preset threshold (e.g., 85%), forming potential reasons for discrepancies. For example, for search results where an interface returns a missing data field, potential reasons for discrepancies might include: 1) The interface response construction code omits the assignment logic for the target field; 2) The data query SQL statement does not include the field; 3) The field definition is mistakenly deleted or commented out in the data model (DTO / entity class); 4) Changes in field serialization rules after framework version upgrades result in the field not being returned correctly; 5) Errors in conditional judgment logic cause the field to only be returned in specific scenarios.
[0043] Meanwhile, the large language model 3 can assign a confidence score to each candidate cause of difference. The scoring criteria include: the frequency of occurrence of the cause in historical regression test root cause cases, the semantic similarity to the current result difference field, and the commonality of the corresponding code change type in the target project. For example, if the omission of field assignment in the response construction code occurs most frequently in the project's historical cases, then the confidence score of this cause is higher, providing a priority reference for the verification process in subsequent step S22.
[0044] The Large Language Model 3 filters candidate reasons for differences based on all result difference fields. The filtered candidate reasons for differences may be one or more, and each candidate reason for difference may correspond to one or more result difference fields.
[0045] In step S22, when determining the code change fragments based on the code change information, the large language model 3 can determine the scope of code modifications in this iteration based on information such as version control logs and change records in the code change information. This scope includes, but is not limited to, the modified file paths, line number ranges, involved function / method names, data structure definitions, configuration parameters, etc., thereby determining all code change fragments. For example, if the version control log shows "modified interface response processing class A", then the code change fragment is the code block corresponding to method B related to response data construction and data field mapping logic in that class; if the change record includes "adjusted method D in numerical calculation utility class C", then the code implementation part of method D is the code change fragment.
[0046] It's important to note that a code change snippet is not necessarily the cause of the result difference. A single task's result difference field can correspond to one or more cause differences, and each cause difference can correspond to one or more result difference fields, or one or more code change snippets. However, a code change snippet does not necessarily correspond to a result difference field or a cause difference.
[0047] After identifying the code change fragments, the large language model 3 sequentially verifies each code change fragment for each candidate reason according to the confidence scores of the candidate reasons in step S21, from high to low, to determine whether each code change fragment meets the characteristics of a candidate reason. This process identifies the code change fragment corresponding to each candidate reason. If a candidate reason does not have a corresponding code change fragment, then that candidate reason does not exist.
[0048] During the verification process, the Large Language Model 3 can combine the programming language syntax rules, the scope of impact of code logic changes, and the data flow / control flow exception pattern library caused by code changes in the target knowledge base 03 to perform matching analysis from three dimensions: code semantics, syntax structure, and logical relevance. For example, for the candidate difference reason "the assignment logic of the target field is missing in the interface response construction code", the Large Language Model 3 analyzes the implementation logic of the corresponding method in the code change fragment and checks whether there is an assignment statement for the missing target field. If the assignment statement is deleted in the code change fragment or the assignment logic of the field is not added, it is determined that the code change fragment has a matching relationship with the candidate difference reason.
[0049] To improve verification efficiency, we can first filter out code change segments with the code characteristics corresponding to the candidate reasons for the difference, and then only verify the code change segments with the characteristics to determine whether the candidate reason for the difference is valid, and if so, which code change segments it corresponds to.
[0050] For example, if the candidate difference reason is "array index out of bounds causing the program to throw an exception", the corresponding code characteristics are array access operations (such as the [] operator, get() method), index value calculation logic, loop boundary condition modification, and array length assignment / change. The Large Language Model 3 can first perform preliminary screening of all code change fragments based on these code characteristics—retaining only fragments containing array access syntax, loop structure modification, or array length-related code, and excluding code change fragments that do not involve array operations (such as simple text description modification, irrelevant configuration parameter adjustment, etc.). This reduces the verification scope from multiple original code change fragments to at least a few target code change fragments, significantly reducing the cost of invalid verification.
[0051] If no code change fragments matching the code characteristics are found after screening, the reason for the candidate difference is directly determined to be invalid, and there is no need to proceed to the subsequent detailed verification stage. If multiple code change fragments matching the characteristics are found, the fragments with a higher correlation to the result difference are verified first (e.g., if the result difference occurs in the order creation time return interface, the code change fragments related to date formatting in the order module are verified first, rather than the date processing fragments in the user module), to further improve the verification efficiency and accuracy.
[0052] In step S23, when the code change segment corresponds to the candidate difference reason, the code change position is located in the code change segment according to the candidate difference reason.
[0053] Among them, a code change fragment refers to the smallest functional unit or syntax block containing code modification operations extracted from version control records (such as Git commits, diff files), covering the complete modification logic (such as a partial implementation of a function, a conditional statement, a set of variable definitions, etc.); a code change location refers to the specific line of code (which can be accurate to the line number) in the code change fragment that directly triggers the candidate difference reason, and is the operation node that leads to the difference in result. A code change fragment can contain at least one code change location (such as multiple lines in the same function triggering the same difference reason, or different lines triggering different difference reasons respectively).
[0054] In some embodiments, a code change segment may correspond to multiple candidate reasons for the difference. If a code change segment corresponds to multiple candidate reasons for the difference, the Large Language Model 3 can locate the differences sequentially according to the confidence level of the reasons for the difference: for example, if a code change segment contains two modifications, "deletion of field assignment logic" and "modification of condition judgment logic", which correspond to two candidate reasons for the difference, namely "missing interface field" and "abnormal field return scenario", then the code line for deleting the assignment logic (e.g., line 89) and the code line for modifying the condition judgment (e.g., line 95) are located as two independent code change locations and associated with the corresponding reasons for the difference.
[0055] In some embodiments, the large language model 3 can directly derive the cause of the difference based on the analysis. If the cause of the difference is not among the recorded candidate causes of difference, the cause of the difference analyzed by the large language model 3 shall prevail.
[0056] If the cause of the difference in the result difference field cannot be determined in the code change fragment, in this embodiment of the disclosure, the call chain information can be used to carry out reverse tracing and correlation investigation to further narrow down the scope of the cause of the difference.
[0057] In an exemplary embodiment, the information acquisition module 2 is further configured to acquire call chain information of the code change fragment, the call chain information including the called method corresponding to the code change fragment and the method that calls the code change fragment.
[0058] In determining the code change location corresponding to the result difference field, the Large Language Model 3 can determine the code change location corresponding to the result difference field based on the call chain information of each code change segment.
[0059] The large language model 3 can call the information acquisition module 2 to obtain the call chain information corresponding to each code change fragment. Based on the call chain information obtained by the information acquisition module 2, it can sort out the complete call chain of the called method corresponding to the code change fragment, as well as the data flow transmission path related to the result difference field. By analyzing the code logic and data interaction relationship of each node in the call chain, it can explore the potential reasons for the difference.
[0060] For example, if the code change fragment is "Data Processing Method A", and direct verification of this fragment does not reveal a reason for the difference corresponding to the result difference field "Order Amount", the Large Language Model 3 parses the call chain information of this code change fragment and determines that Data Processing Method A depends on the upstream "Amount Calculation Method B". Based on this relationship, the Large Language Model 3 checks whether the upstream dependent amount calculation method B has undergone code changes (even if this method was not included in the initial code change fragment set), or whether its return value has been tampered with, type-conversion errors, or other issues when passed to Data Processing Method A.
[0061] Alternatively, if the modified code fragment is called by other methods (i.e. there are downstream caller methods C, D, etc.), and direct verification does not find any association between the modified code fragment and the result difference field, then the large language model 3 traces the downstream call chain based on the call chain information to analyze whether the modification of the modified code fragment indirectly affects the execution result of the downstream method through data transmission or logical dependency, thereby causing the result difference.
[0062] For example, a code snippet might be "date conversion method E in the basic utility class," which modifies the date format from 'yyyy-MM-dd' to 'yyyy / MM / dd'. Direct verification doesn't reveal a direct correlation between this change and the result difference field "order creation time (format error)" (because method E itself doesn't directly participate in order data return). However, through call chain analysis, it's discovered that method E is called by the downstream "order data encapsulation method F" to format the order creation time before returning it to the front end. In this case, the Large Language Model 3 analyzes whether the format change in method E causes the date string received by method F to not conform to the front end's expectations. It then checks whether method F contains secondary processing logic for the date format (such as incompatibility with the new format leading to parsing failure or format conversion errors), or whether method F has a field mapping error when encapsulating the formatted date field into the response object. If the verification reveals that method F is not adapted to the format change of method E, causing the format of the "order creation time" field received by the front end to change from "yyyy-MM-dd" to "yyyy / MM / dd", then it is determined that the code change fragment (format adjustment of method E) indirectly caused the result difference through the downstream call chain. The specific line of code in method E that adjusted the format (such as line 35) is located as the code change location. At the same time, the relevant lines of code in downstream method F that were not adapted (such as the date field encapsulation logic in line 120) are associated as auxiliary change locations to completely restore the chain that caused the difference.
[0063] If the cause of the difference cannot be located by tracing back through the call chain, this embodiment of the disclosure also sets that the large language model 3 can combine the call chain anomaly investigation cases and cross-module data interaction anomaly patterns in the target knowledge base 03 to generate candidate investigation directions (such as "inconsistent parameter passing format of the caller" and "missing return value processing logic of the called method"), and carry out targeted verification for the candidate investigation direction (rather than just limiting the code change fragments).
[0064] Among the code changes that led to the differences in the results, some changes were in line with the expected test objectives (i.e., the requirements of this modification), while others were bugs (e.g., incorrect code).
[0065] Therefore, after determining the code change location that leads to the result difference field, the Large Language Model 3 also determines whether the code change location conforms to the test target information, outputs the target code change location that does not conform to the test target information (i.e. the change line corresponding to the bug) and the target result difference field corresponding to the target code change location, providing developers with a direct basis for fixing the problem.
[0066] In this embodiment of the disclosure, the test target information includes code comments and requirement information corresponding to the code change location. The large language model determines whether the code change location conforms to the test target information by: the large language model obtaining the code comments and requirement information corresponding to the code change location based on the test target information; and the large language model determining whether the result difference field conforms to the expectations of the code comments and requirement information based on the code comments and requirement information.
[0067] The code comments include functional comments for the function / method where the change is located, inline comments (such as / / supplementing the return of the user's mobile phone number field), and commit comments (such as the Git commit message "Optimize the order amount calculation logic to support discount stacking"). The requirement information includes structured or unstructured information such as related requirement document fragments, defect descriptions, and expected test case results. These are obtained by the information acquisition module 2 through parsing code repository comments, associating with the requirement management system interface, and extracting test case documents, and are then synchronized to the large language model 3.
[0068] When determining whether the code change location matches the test target information, the Large Language Model 3 first semantically aligns the modified content (such as field addition / deletion, logic adjustment, parameter change, etc.) with the corresponding code comments and requirement information. For example, if the code change location is line 58, the corresponding comment is " / / hide the last 6 digits of the user's ID number", and the requirement information includes "[Privacy Optimization] De-identify the ID number when returning user details interface", then semantic parsing confirms the core meaning of the three. Figure 1 To (de-identified ID number).
[0069] Next, based on code comments and requirement information, clarify the expected results (such as "the interface returns the desensitized ID number" and "order amount = total price of goods × discount coefficient × membership level coefficient"). Then, combine the code logic and requirement mapping rules and semantic consistency judgment model in the target knowledge base 03 to verify whether the difference fields in the results meet the expectations.
[0070] For example, if the expected test objective is "to return a new user address field", the code change location is "add privateString address; to line 32 of UserDTO.java", and the result difference field is "the interface returns a new address field", then the code change location is determined to be consistent with the test objective, and the result difference is an expected change; if the expected test objective is "to optimize the accuracy of amount calculation and retain two decimal places", the code change location is "to change BigDecimal.setScale(2) to setScale(0) to line 45 of MathUtil.java". If the code change results in the "Order Amount" field only retaining integer digits, then by comparing the expected (2 decimal places) with the actual difference (integer digits), it is determined that the code change location does not conform to the test target information and is a bug. If the code comment is "Fix array index out of bounds problem", and the requirement information is "[Defect Fix] An exception is thrown due to incorrect array length judgment when importing data in batches", and the code change location is the corresponding code, but it actually increases the risk of index out of bounds, then the Large Language Model 3 determines that the change location does not conform to the test target information and is a logical error bug by analyzing the expected (solving index out of bounds) of the comment and the actual impact of the code modification (exacerbating the exception).
[0071] In some embodiments, if code comments or requirement information are ambiguous (e.g., the comment "adjust parameter configuration") or ambiguous (e.g., the requirement "optimize performance" can correspond to multiple implementations), the Large Language Model 3 can combine historical requirement-change mapping cases and business domain common sense in the target knowledge base 03 to supplement contextual information and resolve ambiguities. For example, if the requirement information is "optimize interface response speed" and the code change location is "delete redundant database query operations on line 63 of QueryUtil.java", combined with the case in the knowledge base that "deleting redundant queries can improve interface performance", it is determined that the change meets the test objective; if the code change location is "add loop traversal logic", and there is no other supplementary information indicating that the traversal is a necessary operation for performance optimization, then it is determined that the change does not match the test objective information and further prompts for manual confirmation are required.
[0072] Finally, the large language model 3 outputs structured judgment results, such as: The target code was changed at line 45 of MathUtil.java; Changes: BigDecimal.setScale(2) → setScale(0); Related target result difference field: Order amount (precision lost, only integer is retained); Judgment result: Does not meet the test target information (expected to retain 2 decimal places); Risk level: High.
[0073] In addition, it can be accompanied by judgment criteria (code comments, requirement information snippets, semantic comparison results) to ensure that developers can quickly understand the cause and scope of the bug.
[0074] In addition to locating the code change that caused the difference in the results, in some embodiments, the cause of the difference in the results also includes configuration modification. That is, configuration modification may cause one or more difference in the results, and code change is not the cause of these difference in the results; or, configuration modification and code change may both be the cause of the difference in the results.
[0075] In an exemplary embodiment, the large language model 3 is also used to automatically determine the configuration change location corresponding to the result difference field based on the test target information, code change information, and configuration change information.
[0076] Determining the location of configuration changes can be done simultaneously with or after determining the location of code changes. For example, if after verifying all code change fragments, the location of the code change causing the result difference field still cannot be determined—that is, the reason for the association between the code change location and the result difference field cannot be confirmed through code logic analysis, call chain tracing, etc.—or if all code change fragments are verified to meet the test target information (no code-level bugs), then the Large Language Model 3 automatically switches to configuration change investigation mode to find the configuration change location causing the aforementioned result difference field.
[0077] For example, in some complex business scenarios, code changes and configuration changes may have a synergistic effect. In this case, the Big Language Model 3 can simultaneously start the process of locating the code change location and the configuration change location. On the one hand, it verifies the matching relationship between the code change fragment and the candidate difference reasons according to the aforementioned steps to locate the change location at the code level; on the other hand, it simultaneously filters candidate configuration items related to the result difference fields to verify whether the configuration change meets the test objectives.
[0078] For example, if the result difference field is "user points redemption ratio changed from 100:1 to 200:1", and the test target information is "points redemption ratio adjusted to 150:1", when investigating code changes, the Large Language Model 3 verifies whether the points calculation logic has been modified (such as a hard-coded adjustment of the redemption ratio); simultaneously, when investigating configuration changes, it searches for change records of relevant configuration items such as "point.exchange.rate". If it is found that the code has not modified the redemption ratio logic, but the configuration item "point.exchange.rate" has been changed from 100 to 200, and the points redemption logic in the code is already configured to read this parameter, then the configuration change location (corresponding to line 45 of the configuration file) can be simultaneously determined as the direct cause of the difference, and the code has correctly adapted to the configuration reading logic, without the need for additional code change location.
[0079] If parallel investigation reveals that code changes and configuration changes both contribute to the difference in results, the Large Language Model 3 can output structured information on the location of code changes and the location of configuration changes, and clarify the relationship and influence weight between the two.
[0080] In the exemplary embodiment, when searching for the configuration change location that causes the above-mentioned result difference field, the large language model 3 first determines the mapping relationship between configuration items and business functions and the difference cases caused by historical configuration changes based on the target knowledge base 03, and establishes a preliminary association between the result difference field and the configuration item, and filters out the range of candidate configuration items that may affect the result difference field, based on the test target information synchronized by the information acquisition module 2 (such as configuration adjustment requirements and expected configuration effects in the requirements document), code change information (code modification records of this iteration, used to exclude the impact at the code level), configuration change information (including structured data such as configuration file path, changed configuration item key, values before and after the change, change time, operator, etc., covering various configuration forms such as local configuration files, configuration center parameters, and environment variables).
[0081] Next, analyze the code change information to determine if there are any code logic modifications related to the difference in the results. If no related code modifications have been made, or the code modification is only the adaptation logic for "reading configuration items" (without changing the effective rules of the configuration itself), then the configuration change information is verified; if there are linked modifications between the code and the configuration, then the synergistic effect of the two is simultaneously verified to ensure that it meets the test objectives.
[0082] Next, the configuration change records corresponding to the candidate configuration items will be checked one by one. Combined with the clearly defined expected configuration values / effects in the test target information, it will be determined whether the configuration change is the direct or indirect cause of the result difference: (1) If there are no relevant modifications to the code, and there is a direct mapping relationship between the value before and after the change of a certain configuration item and the result difference field, and it does not meet the test target information, then the change location of the configuration item is located as the configuration change location, and the configuration change is determined to be the only reason for the difference.
[0083] (2) If the code modifies the reading logic of the configuration item (such as adding multi-environment configuration reading or configuration parameter conversion), and the configuration item itself is changed, both of which together cause the difference in results, then locate the location of the configuration change and the location of the related code change respectively, and clarify the synergistic effect of the two.
[0084] (3) If the combination of changes to multiple configuration items causes differences in results, locate the change location of each configuration item and analyze the combined impact.
[0085] Once the location is established, the Large Language Model 3 can output a structured result containing "configuration file path + specific line number + configuration item key + values before and after the change + fields showing differences in related results + matching status with the test target + scope of impact", providing guidance for configuration rollback or adjustment.
[0086] For example: Configuration change location: line 123 of / config / prod / application.yml; Configuration key: order.prod.timeout; Before the change: 5 minutes; After the change: Not configured (default 5 minutes will be used); Related result difference field: Order timeout time (5 minutes); Matching result: Does not meet test objective (expected 10 minutes); Scope of impact: All order creation processes in the production environment.
[0087] In some embodiments, the large language model 3 is further configured as follows: Based on the test target information, code change information, and configuration change information, analyze the logical reasons for the differences in the output results and provide suggestions.
[0088] The Large Language Model 3 can be configured to output a logical analysis and suggestions regarding the reasons for the differences, along with the location of changes to the target code and / or the target configuration. Alternatively, it can output a logical analysis and suggestions regarding the reasons for the differences when the location of the changes to the target code and the configuration cannot be determined.
[0089] The logical analysis of the reasons for the differences may include, for example, the overall differences between the result difference field and the test target information, the reasoning summary of the reasons for the result difference field and the evidence chain to exclude other reasons, the explanation of the technical principles of the changes that caused the differences, as well as the assessment of the scope of impact, the classification of risk levels and the provision of recommendations.
[0090] In some embodiments, the system 100 further includes a self-training module, which is used to obtain human feedback results corresponding to the target result difference field and the target code change location, and optimize the input prompts of the large language model or fine-tune the model parameters based on the human feedback results.
[0091] After the target code and / or target configuration are changed in the output of the large language model 3, the accuracy of the model's investigation results and reasons can be judged manually. Inaccurate cases are analyzed regularly and human feedback results are generated to build a feedback dataset. The self-training module uses this feedback dataset to regularly fine-tune the large language model's prompt and experience, forming a feedback-optimization closed loop to continuously improve the model's capabilities.
[0092] Figure 3 This is a schematic diagram of the execution logic for root cause localization of differences in regression test results in an exemplary embodiment of this disclosure.
[0093] refer to Figure 3 In phase 301, system 100 obtains code change information, test target information, and configuration change information by calling the code change interface and configuration change log; it connects to the test platform to obtain the diff field (result difference field), summarizes and deduplicates the fields from multiple requests, and only retains inconsistent values.
[0094] In phase 302, system 100 uses the RAG target knowledge base's external extended knowledge, built-in investigation strategies, and investigation experience to conduct retrieval. Based on the retrieval results, it locates the root cause of the difference fields in the results, provides the root cause code / configuration that caused the change, interprets the results, and outputs the analysis logic.
[0095] In phase 303, System 100 can also obtain call chain information corresponding to the code change fragment, and use this information to locate the code change location. For complex scenarios (field names and logic undergoing multiple transformations, changed values, change sources being configuration / data rather than code, etc.), the cause cannot be located solely based on the code change fragment itself, and the full codebase is too lengthy. Therefore, analyzing the logical relationship between the code change fragment and the entire call chain to locate the root cause requires combining the call chain information corresponding to the code change fragment to obtain code fragments of upstream and downstream methods for analysis. Call chains can be obtained using open-source tools such as call_graph.
[0096] In phase 304, System 100 obtains the code comments and requirement information corresponding to the code change location. Based on the code comments and requirement information, it determines whether the result difference fields meet the expectations of the code comments and requirement information, and outputs the judgment result and the basis for the judgment. Specifically, it can query code comments, collect requirement documents and detailed design documents, and after understanding the purpose of the requirement, the goal of the technology upgrade, the meaning of the fields and the functions, it gives the judgment conclusion and basis for whether it meets the expectations.
[0097] In phase 305, the adoption results are manually labeled. Users can manually evaluate the investigation results and reasons provided by the system, periodically analyze inaccuracies, and adjust the large model prompt and experience to form a feedback-optimization closed loop, continuously improving the model's capabilities.
[0098] In an exemplary embodiment of this disclosure, an electronic device capable of performing some or all of the functions of the system 100 described above is also provided.
[0099] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”
[0100] The following reference Figure 4 To describe an electronic device 400 according to this embodiment of the present invention. Figure 4 The electronic device 400 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0101] like Figure 4 As shown, the electronic device 400 is presented in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processor 410, at least one memory 420, and a bus 430 connecting different system components (including memory 420 and processor 410).
[0102] The memory stores program code that can be executed by the processor 410, causing the processor 410 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processor 410 can perform the methods shown in the embodiments of this disclosure.
[0103] The memory 420 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 4201 and / or cache 4202, and may further include read-only memory (ROM) 4203.
[0104] The memory 420 may also include a program / utility 4204 having a set (at least one) of program modules 4205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0105] Bus 430 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.
[0106] Electronic device 400 can also communicate with one or more external devices 500 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 400, and / or with any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. As shown, network adapter 460 communicates with other modules of electronic device 400 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0107] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0108] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.
[0109] The program product for implementing the above-described method according to embodiments of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0110] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0111] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.
[0112] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0113] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0114] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0115] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0116] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and concept of this disclosure are indicated by the claims.
Claims
1. A system for locating the root causes of differences in regression test results, characterized in that, include: The test result acquisition module connects to the test platform and is used to obtain actual test results and online version test results; The information acquisition module connects to the development platform and is used to call the code change interface of the development platform to obtain code change information and test target information associated with the actual test results. A large language model, connected to a target knowledge base, is used to automatically determine the result difference fields based on the actual test results and the online version test results, based on the semantic understanding and logical reasoning of the code change information. It also determines the code change location corresponding to the result difference fields and whether the code change location conforms to the test target information based on the test target information and the code change information. Finally, it outputs the target code change location that does not conform to the test target information and the target result difference fields corresponding to the target code change location.
2. The regression test result difference root cause localization system as described in claim 1, characterized in that, The information acquisition module is also used to call the configuration change log of the development platform to obtain configuration change information associated with the actual test results; The large language model determines the code change location corresponding to the result difference field and whether the code change location conforms to the test target information based on the test target information, code change information, and configuration change information. It then outputs the target code change location that does not conform to the test target information and the target result difference field corresponding to the target code change location.
3. The regression test result difference root cause localization system as described in claim 1, characterized in that, The large language model automatically determines the code change location corresponding to the result difference field based on the test target information, the code change information, and the configuration change information, including: The search keywords are determined based on the result difference field, and the target knowledge base is searched based on the search keywords to determine the reasons for the differences. The search keywords include any one or a combination of the name, type, and interface path of the result difference field. Based on the code change information, code change segments are determined, and each code change segment is verified based on the candidate reasons for difference to determine whether the code change segment meets the candidate reasons for difference. When the code change segment corresponds to the candidate difference reason, the code change position is located in the code change segment according to the candidate difference reason.
4. The regression test result difference root cause localization system as described in any one of claims 1 to 3, characterized in that, The information acquisition module is also used to acquire the call chain information of the code change fragment, the call chain information including the called method corresponding to the code change fragment and the method that calls the code change fragment; The large language model determines the code change location corresponding to the result difference field based on the test target information and the code change information, including: The code change location corresponding to the result difference field is determined based on the call chain information corresponding to the code change fragment.
5. The regression test result difference root cause localization system as described in claim 1, characterized in that, The test target information includes code comments and requirement information corresponding to the code change location. The large language model determines whether the code change location conforms to the test target information by: The large language model obtains the code comments and requirement information corresponding to the code change location based on the test target information; The large language model determines whether the result difference field matches the expectations of the code comments and requirement information based on the code comments and requirement information.
6. The regression test result difference root cause localization system as described in claim 1, characterized in that, The large language model is also used for: Based on the test target information, the code change information, and the configuration change information, the system automatically determines the target configuration change location that does not conform to the test target information corresponding to the result difference field.
7. The regression test result difference root cause localization system as described in claim 1, characterized in that, The large language model is also used for: Based on the test target information, the code change information, and the configuration change information, output the logical analysis and suggestions for the differences in the result difference fields.
8. The regression test result differential root cause localization system as described in claim 1, characterized in that, The large language model receives test identification information input manually, calls the test result acquisition module and the information acquisition module to obtain the actual test results and online version test results corresponding to the test identification information, code change information associated with the actual test results, test target information and configuration change information, and outputs the target code change location that does not conform to the test target information and the target result difference field corresponding to the target code change location based on the actual test results and online version test results corresponding to the test identification information, code change information associated with the actual test results, test target information and configuration change information.
9. The regression test result difference root cause localization system as described in claim 1, characterized in that, Also includes: The self-training module is used to obtain the human feedback results corresponding to the target result difference field and the target code change location, and optimize the input prompts of the large language model or fine-tune the model parameters based on the human feedback results.
10. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being configured to run the test result difference field localization system as described in any one of claims 1-9 based on instructions stored in the memory.