Automatic test case generation method and system based on code change semantic analysis

Abstract

The invention provides an automatic test case generation method and system based on code change semantic analysis, computing equipment and a computer readable storage medium. In the method, when a user submits codes to a code hosting platform, code differences are obtained through a monitoring function; automatically extracting a changed function context through a code analyzer according to code differences, and analyzing a cross-file and cross-module calling relationship; in a test case generation and execution link, a standard test case format is output through a large language model, and an existing automatic test framework is automatically docked; besides, automatic and standardized test suggestion generation is realized through a customized large language model, and clear and definite test key points, risk prompts and test method suggestions are automatically output in combination with business contexts such as a historical defect library, a project background and a user scale; complete analysis records, case data, execution results and defect information are uniformly stored in a knowledge base to serve as basic data for continuous training of the large language model, and continuous optimization and iteration of the model are supported.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular relates to a code change semantic analysis-based automatic test case generation method and system, a computing device, and a computer readable storage medium. BACKGROUND

[0002] With the increasing complexity of enterprise-level game, financial, e-commerce, and other large project development processes, software version iteration is frequent, and operations such as requirement change, bug fixing, and performance optimization have a large number of code submissions every day. The traditional testing method mainly relies on manual analysis of change content and manual writing of test cases, which has the following pain points: high analysis complexity, the diff fragment submitted by the developer is often only a partial code fragment, and it is difficult to directly see its overall impact on the functional context and business; it takes a lot of time from code submission to manual writing and execution of test cases, and the opportunity to find problems is lagging behind; manual analysis is highly subjective and easy to miss low-frequency paths and abnormal scenarios. Therefore, an automatic test case generation method is needed to solve the above problems. SUMMARY

[0003] Therefore, the present application provides a code change semantic analysis-based automatic test case generation method, system, computing device, and computer readable storage medium to solve the technical defects in the prior art.

[0004] According to a first aspect of an embodiment of the present application, a code change semantic analysis-based automatic test case generation method is provided, comprising:

[0005] When the code is submitted, the code difference is obtained through the listening function of the code hosting platform;

[0006] The code difference is parsed to obtain the context information of the modified function;

[0007] The call relationship of the modified function is analyzed to identify the affected range of the modified function;

[0008] The test case is automatically generated according to the context information and the affected range of the modified function.

[0009] According to a second aspect of an embodiment of the present application, a code change semantic analysis-based automatic test case generation system is provided, comprising:

[0010] A listening unit is configured to obtain a code difference through a listening function of a code hosting platform when the code is submitted;

[0011] A first parsing unit is configured to parse the code difference to obtain context information of a modified function;

[0012] A second analysis unit is configured to analyze the calling relationship of the modified function and identify an affected range of the modified function.

[0013] A generation unit is configured to automatically generate a test case based on the context information and the affected range of the modified function.

[0014] According to a third aspect of the embodiments of the present application, a computing device is provided, which includes a memory, a processor, and computer instructions stored in the memory and executable on the processor, and the processor implements the steps of the method for automatically generating a test case based on semantic analysis of code changes.

[0015] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer instructions for implementing the steps of the method for automatically generating a test case based on semantic analysis of code changes.

[0016] In the embodiments of the present application, when a user submits code to a code hosting platform, first, the code differences are obtained through a listening function. Further, the complete function body information of changes is automatically extracted by a code parser according to the code differences, and the calling relationship across files and modules is analyzed to accurately identify the affected range, thereby effectively reducing false positives and false negatives. In the generation and execution of test cases, the present application outputs a standard test case format through a large language model, and automatically interfaces with existing automated testing frameworks. By calling existing cloud real machine platforms and interface testing frameworks, automatic scheduling and execution of test cases are realized without human intervention.

[0017] In addition, the present application also realizes automatic and standardized test suggestion generation through a customized large language model, automatically outputs clear and specific test priorities, risk prompts, and test method suggestions in combination with historical defect libraries, project backgrounds, user scales, and other business contexts; and stores complete analysis records, test case data, execution results, and defect information in a knowledge base as basic data for continuous training of the large language model, supports continuous optimization and iteration of the model, and makes it more and more suitable for actual projects. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a structural block diagram of a computing device provided by the embodiments of the present application;

[0019] Figure 2 is a flowchart of a method for automatically generating a test case based on semantic analysis of code changes provided by the embodiments of the present application;

[0020] Figure 3 is a structural diagram of a system for automatically generating a test case based on semantic analysis of code changes provided by the embodiments of the present application. DETAILED DESCRIPTION

[0021] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure aspects of the present application.

[0022] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to limit one or more embodiments of the present application. As used in one or more embodiments of the present application and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal sequence, but are used only to distinguish one from another. For example, without departing from the scope of one or more embodiments, first can be termed second, and similarly, second can be termed first. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "responsive to the determination" or "in response to the determination."

[0024] In the present application, an automatic test case generation method and system based on code change semantic analysis, a computing device and a computer readable storage medium are provided, which are described in detail one by one in the following embodiments.

[0025] Figure 1 A structural block diagram of a computing device 100 according to an embodiment of the present application is shown. The components of the computing device 100 include, but are not limited to, a memory 110 and a processor 120. The processor 120 is connected to the memory 110 through a bus 130, and a database 150 is used to save data.

[0026] The computing device 100 also includes an access device 140 that enables the computing device 100 to communicate via one or more networks 160. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 140 can include one or more of any type of network interface (for example, a network interface card (NIC)), wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and so on.

[0027] In one embodiment of the present application, the above-described components of the computing device 100, as well as other components not shown in FIG. 1, can be connected to each other by a bus, for example. Figure 1 It should be appreciated that the computing device structure diagram shown is merely for the purpose of example, and is not a limitation on the scope of the present application. Other components can be added or replaced as needed by those skilled in the art. Figure 1 It should be appreciated that the computing device structure diagram shown is merely for the purpose of example, and is not a limitation on the scope of the present application. Other components can be added or replaced as needed by those skilled in the art.

[0028] The computing device 100 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and so on), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and so on), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 100 can also be a mobile or stationary server.

[0029] In the prior art, the test analysis after code changes mainly relies on the following technical means:

[0030] One is the line-level difference detection tool represented by Git diff, SVN diff and the like. These tools can identify which files and lines have been changed, but cannot understand based on the syntax structure which function, class or module these changes belong to, nor can they judge the impact range of the code changes on the overall system function. They only provide the most basic difference information, and the tester still needs to manually analyze the business significance and potential impact of these changes, which is inefficient and easy to miss risk points.

[0031] Another type is static code analysis tools, such as SonarQube, Pylint, and Flake8. These are typically used to detect issues like code style and potential syntax errors. While these tools can assist in quality assurance to some extent, they focus more on code quality itself and do not understand the business semantics of the code. They also cannot provide test suggestions for functional changes, nor can they automatically generate test cases based on changes.

[0032] In addition, some AI-assisted development tools (such as Github Copilot and Diffblue) have certain code understanding capabilities, but their functions are mainly focused on code completion and unit test generation. They lack the ability to perform semantic analysis of code changes and automatically generate business test cases, and they have not formed a complete closed loop from "change-risk analysis-test case generation-execution verification".

[0033] Therefore, in order to overcome the limitations and defects in the prior art, an automated test case generation method and system based on code change semantic analysis is proposed in the embodiments of this application.

[0034] Instruction manual attached Figure 1 The processor 120 in the middle can execute Figure 2 The steps in an automated test case generation method based on code change semantic analysis are shown. Figure 2 The flowchart shows a method for generating automated test cases based on code change semantic analysis, including steps 202 to 206.

[0035] Step 202: When submitting code, obtain code differences through the monitoring function of the code hosting platform.

[0036] In this step, a WebHook is configured for the code hosting platform, such as GitLab or GitHub, so that the WebHook points to an address on the local machine or server. Whenever code is committed, information about changes to the entire project is pushed to this address.

[0037] Furthermore, after a Webhook event is triggered, the receiving server needs to process the request to obtain detailed information about the code changes, including code diffs.

[0038] Specifically, the Receiver is a service or API endpoint that accepts HTTP requests (usually in JSON format) from GitLab. The requests contain event type identifiers and related information (such as commit ID, branch information, etc.).

[0039] The server receives and parses JSON data to extract valuable information, such as:

[0040] repository: code repository information.

[0041] commits: commit list containing changes.

[0042] ref: reference (such as a branch) of the target of the change.

[0043] If code difference data is needed, the commit ID can be combined to further obtain detailed difference information through the API of GitLab:

[0044] Send a GET request to the REST API endpoint of GitLab. For example:

[0045] GET / projects / :id / repository / commits / :sha / diff

[0046] Further, the file list and specific line code difference of the change are obtained from the response returned by the above API. For example:

[0047]

[0048] Store the obtained difference data in the local system or intermediate database for subsequent analysis and processing.

[0049] Step 204: Analyze the code difference to obtain the context information of the modified function.

[0050] In this step, the target source code is parsed to build a syntax tree, and the information of the modified function is identified according to the obtained code difference, including function name, location, and code context, etc.

[0051] In one possible implementation, the entire target source code file such as main.py in step 202 is parsed by the tree-sitter tool to generate an AST syntax tree, or the call area of a specific code segment is tracked.

[0052] Further, the function_definition node is traversed from the generated syntax tree to identify the modified part, including: comparing the modified line number provided by the diff with the function start and end line number in the syntax tree, finding the function node, and then judging whether it is the modified part by the intersection of the line number span and the modified line. For example: through Git diff, it is known that the 2nd to 4th lines of the main.py file are modified, and when the tree-sitter creates a syntax tree that covers these lines, the add function is identified as the modified function.

[0053] Further, the function definition of the modified function, meta information (such as the complete boundary of the function, decorator, annotation, parameter, return value type, etc.) and the class definition where the modified function is located, module information, etc. are identified. Among them, the extraction of the above information is completed by identifying different semantic nodes from the syntax tree, including function definition (function_definition), class definition (class_definition) and module level (module) information.

[0054] Step 206: Analyze the inter-function call relationship to identify the affected scope of the modified function.

[0055] In this step, the information obtained in step 204 is analyzed, and the call relationship of the modified function in the project is tracked to return the call position and the module where the call is located, which is used to identify which part of the project is affected.

[0056] In a feasible implementation, all call positions of the modified function in the project are obtained by a static analysis tool.

[0057] Specifically, the import path is parsed by a static analysis tool (such as Jedi), and the source of the function and class is located. In a multi-file and multi-module project, functions and classes are referenced through import statements. In order to accurately analyze the call relationship, it is necessary to correctly parse these import paths, such as:

[0058] Absolute path import: use the complete path, such as import my_package.my_module.

[0059] Relative path import: use. to represent the relative path, such as from.my_module import my_function.

[0060] Cross-file parsing: parse and track these imports to identify the actual source of function and class definitions.

[0061] Further, the static analysis tool can also handle complex module structures, including:

[0062] Package structure: handle symbols exported in __init__.py.

[0063] Multi-level module: support nested submodule calls, such as my_package.sub_package.module.

[0064] Path resolution: automatically resolve relative and absolute paths to make function and class call analysis more accurate.

[0065] Further, the ability to identify the following call types is achieved through a static analysis tool:

[0066] Normal function call: Direct call form func().

[0067] Class method call:

[0068] • Instance method: instance.method().

[0069] • Class method: cls.method() or Class.method().

[0070] • self call: Call within a class with self.method().

[0071] Dynamic import and decorator wrapping

[0072] • Dynamic import: e.g. importlib or __import__ for dynamically loading modules, analyze dynamic imports by tracing symbol references.

[0073] • Decorator wrapping: Decorators can change the behavior of functions and sometimes wrap function calls, parse decorator chains to identify underlying functions.

[0074] Through static analysis tools, it is possible to effectively track the cross-file call relationship of functions and classes in complex projects. Not only can it analyze all types of import paths, including absolute paths and relative paths, but it can also identify various calling methods, such as normal function calls, class method calls, and dynamic imports and decorator wrapping functions. This makes it possible to build detailed function call graphs, such as the call graph of the add function:

[0075]

[0076] Among them, the Defined in position is used to show the file where the function is defined, such as package / math_utils.py.

[0077] The Called in position is used to show the specific file and line number where the call is made, such as main.py at line 6.

[0078] Additional Info: Provides context information for the call, such as the function's purpose (calculating the total price, performing unit testing), etc.

[0079] Step 208: Automatically generate test cases based on the context information of the modified function and the affected range.

[0080] In this step, the context information of the modified function and the affected range, including the modules and functions directly or indirectly affected by the modification, are input into the LLM large language model trained using the history problem library to automatically generate test cases.

[0081] In a feasible implementation, the LLM is used to generate detailed test case information (input, expected output, test steps), and the output is automatically formatted into YAML.

[0082] In a feasible implementation, the LLM model translates the analyzed test case logic into pytest cases, including the use of fixtures and assertions. The output Python code is compatible with existing code libraries and can be directly imported into the project for testing.

[0083] In a feasible implementation, the LLM model outputs the analyzed test case logic into scripts in a specific format for interface operation or device interaction. The scripts can directly call cloud platform devices for real-time testing, ensuring the authenticity and reliability of the testing process.

[0084] Preferably, a large language model (LLM) is used to generate risk points and test recommendations. In order to more comprehensively understand the impact of code modification and develop effective testing strategies, the LLM is guided to generate more business-oriented test recommendations through specific prompts and the output content of steps 204 and 206.

[0085] Specifically, in a feasible implementation, the role is defined, and when interacting with the LLM, the model is prompted to think from the perspective of a senior test specialist. This role setting helps guide the model to focus on advanced concepts related to software quality, such as usability, reliability, and user experience. The LLM is guided to focus on business risks in project operation, rather than just technical implementation.

[0086] Furthermore, the test recommendations output by the LLM include:

[0087] Recommended test types (API / UI / unit test): According to business requirements and function implementation, recommend appropriate test types. For example, critical business logic updates require intensive API testing; parts with frequent user interaction require UI testing.

[0088] Key test scenarios: The LLM output should specify which specific use scenarios need special attention, such as abnormal data input, stability under load, core transaction operations, etc.

[0089] Potential risk paths: Indicate code logic paths or user operation paths that may cause risks. For example, certain unstable dependent resources may affect user experience, or certain error handling may cause data loss.

[0090] Test data design suggestions: Provide guidance on how to design effective test data to cover various boundary conditions, extreme cases, and normal operations. For example, design complex combination inputs to verify whether the system handles all cases reasonably.

[0091] Preferably, code changes, test suggestions, test cases, execution results, and actual bugs are archived together, and large language models (LLM) are periodically trained or fine-tuned, enabling the model to learn and adapt to changes from constantly updated business data, truly achieving dynamic intelligence.

[0092] Periodic training or fine-tuning of large language models includes:

[0093] Periodically collect and process newly archived business data, adjust model parameters by adding the latest cases, and continuously learn new trends and problem-solving methods.

[0094] Fine-tune LLM with domain-specific data to ensure that semantic context, use case generation logic, risk judgment, and other parts better meet domain requirements. This allows the LLM model to treat each analysis as an opportunity to acquire new business knowledge, and to implant diverse business scenarios and problem-solving paths during the training process. By continuously updating and adjusting the model, we ensure that the analysis capability keeps pace with the times.

[0095] In embodiments of the present application, when a user submits code to a code hosting platform, first obtain the code differences through the listening function. Further, through parsers such as Tree-sitter and AST, automatically extract the complete function body information of the changes according to the code differences, and analyze the call relationship across files and modules to accurately identify the affected scope, effectively reducing false positives and false negatives.

[0096] In the generation and execution of test cases, the present application outputs standard test case formats such as YAML, JSON, Python scripts, and Robot Framework scripts through large language models, automatically interfaces with existing automated testing frameworks. By calling existing cloud real machine platforms and interface testing frameworks, it realizes the automatic scheduling and execution of test cases without human intervention.

[0097] In addition, the present application also realizes the automatic and standardized generation of test suggestions through customized large language models, automatically outputs clear and specific test priorities, risk prompts, and test method suggestions in combination with historical defect libraries, project backgrounds, user scales, and other business contexts; and stores complete analysis records, test case data, execution results, and defect information in the knowledge base as basic data for continuous training of the large language model, supporting the continuous optimization and iteration of the model, making it more and more in line with the actual project.

[0098] Corresponding to the method embodiments described above, the present application also provides an embodiment of an automatic test case generation system based on code change semantic analysis, as shown in Figure 3 The system comprises:

[0099] A monitoring unit is configured to obtain code differences through a monitoring function of a code hosting platform when the code is submitted.

[0100] A first analysis unit is configured to analyze the code differences to obtain context information of a modified function.

[0101] A second analysis unit is configured to analyze the calling relationship of the modified function and identify the affected range of the modified function.

[0102] A generation unit is configured to automatically generate test cases according to the context information and the affected range of the modified function.

[0103] The above is a schematic scheme of an automatic test case generation system based on code change semantic analysis according to the embodiment. It should be noted that the technical scheme of the automatic test case generation system based on code change semantic analysis belongs to the same concept as the technical scheme of the automatic test case generation method based on code change semantic analysis described above. The details of the technical scheme of the automatic test case generation system based on code change semantic analysis that are not described in detail can be referred to the description of the technical scheme of the automatic test case generation method based on code change semantic analysis described above.

[0104] In an embodiment of the present application, a computing device is also provided, which comprises a memory, a processor, and computer instructions stored in the memory and executable on the processor. When the processor executes the instructions, the steps of the automatic test case generation method based on code change semantic analysis are implemented.

[0105] The above is a schematic scheme of a computing device according to the embodiment. It should be noted that the technical scheme of the computing device belongs to the same concept as the technical scheme of the automatic test case generation method based on code change semantic analysis described above. The details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the automatic test case generation method based on code change semantic analysis described above.

[0106] In an embodiment of the present application, a computer readable storage medium is also provided, which stores computer instructions. When the instructions are executed by a processor, the steps of the automatic test case generation method based on code change semantic analysis described above are implemented.

[0107] The above is a schematic solution of the computer readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the automatic test case generation method based on code change semantic analysis described above belong to the same concept, and the technical solution of the storage medium which is not described in detail can be seen from the description of the technical solution of the automatic test case generation method based on code change semantic analysis.

[0108] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still accomplish desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

[0109] The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0110] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0111] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be seen from the related description of other embodiments.

[0112] The preferred embodiments of the application disclosed above are only to facilitate the explanation of the application. Alternative embodiments do not describe all the details and do not limit the application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the application. The application selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and utilize the application. The application is limited by the claims and their full scope and equivalents.

Claims

1. A method for automated test case generation based on code change semantic analysis, characterized in that, The method comprises the following steps: obtaining code differences through a monitoring function of a code hosting platform when the code is submitted; parsing the code differences to obtain context information of a modified function; analyzing a calling relationship of the modified function to identify an affected range of the modified function; generating a test case automatically according to the context information and the affected range of the modified function.

2. The method according to claim 1, wherein, The step of obtaining the code differences through the monitoring function of the code hosting platform comprises the following steps: configuring a WebHook of the code hosting platform, and obtaining detailed information of code changes by a receiving server after a WebHook event is triggered.

3. The method according to claim 1, wherein, The step of parsing the code differences to obtain the context information of the modified function comprises the following steps: analyzing a calling area of a target source code file or a specific code segment to generate an AST syntax tree; traversing a function definition node in the generated AST syntax tree, and comparing and identifying a modified code part to obtain a complete function body of the modified function; identifying function definition and function metadata of the modified function, and class definition and module node information in which the modified function is located.

4. The method of claim 1, wherein, The step of analyzing the calling relationship of the modified function to identify the affected range of the modified function comprises the following steps: obtaining all calling positions of the modified function in a project by a static analysis tool, and constructing a function call graph of the modified function.

5. The method of claim 1, wherein, The step of generating the test case automatically according to the context information and the affected range of the modified function comprises the following steps: inputting the context information and the affected range of the modified function into a large language model trained using a historical problem library to automatically generate the test case.

6. The method of claim 5, wherein, The step of generating the test case automatically according to the context information and the affected range of the modified function further comprises the following steps: generating risk points and test suggestions simultaneously by using the large language model; archiving code changes, test suggestions, test cases, execution results, and actual bugs uniformly; collecting and processing new archived business data regularly, and training or fine-tuning the large language model according to the business data.

7. An automated test case generation system based on code change semantic analysis, characterized by, The method comprises the following steps: a monitoring unit is configured to obtain code differences through a monitoring function of a code hosting platform when the code is submitted; a first parsing unit is configured to parse the code differences to obtain context information of a modified function; a second parsing unit is configured to analyze a calling relationship of the modified function to identify an affected range of the modified function; a generating unit is configured to generate a test case automatically according to the context information and the affected range of the modified function.

8. A computing device comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, wherein, The processor executes the instructions to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium storing computer instructions, wherein, The instructions are executed by the processor to implement the steps of the method according to any one of claims 1-6.

Images