Software testing method and system based on large model

Abstract

The invention discloses a software testing method and system based on a large model, and relates to the technical field of software testing, and the method comprises the following steps: step 1: describing a code review rule, and expanding specifications required to be checked: firstly, listing the check specifications required to be satisfied by a code, and according to a thinking chain and a zero- / feed-shot example, establishing a list of the check specifications required to be satisfied by the code; the check rules in the list are described as reasoning instructions of the model according to the check key points and the check steps through combination of one or more technologies of context learning; 2, code slicing: performing structure analysis and dependency analysis on the tested software project: firstly, slicing the code according to a function by utilizing a code analysis tool; according to the software testing method and system based on the large model, convenient project management, user-defined configuration and high-automation-degree test execution can be implemented on a software evaluation task, and the evaluation efficiency is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a software testing method and system based on a large model. Background Art

[0002] Software testing is a key process in software development, which involves evaluating the quality of software products. The main goal of software testing is to identify defects or errors in the software system that may cause incorrect or unexpected behavior. The contradiction between the large scale and complexity of software and the requirements of high quality and high efficiency is becoming increasingly prominent, which has brought unbearable pressure to software testing. At the same time, due to the development of networking, multi-core parallelization and intelligent software, the complexity of software systems and their verification cycles are gradually increasing. The existing traditional software testing methods and systems are difficult to meet the growing needs of complex embedded software evaluation.

[0003] In recent years, the development of artificial intelligence technology, especially large language models, has provided new possibilities for intelligent software testing. Large models have powerful natural language processing and deep learning capabilities, and can understand and generate complex code structures, thus playing an important role in software testing. By combining the intelligent analysis capabilities of large models, automated, efficient and accurate code review can be achieved, significantly improving the efficiency and quality of software testing.

[0004] In the field of intelligent software testing, large language models are widely used in the middle and late stages of the software testing life cycle. For example, in the test case preparation stage, large language models have been used to generate unit test cases, test predictions, and system test inputs. These tasks are crucial in the mid-term stage of software testing and help to discover and resolve problems in advance.

[0005] Secondly, in the defect inspection phase, large models are also used to detect poor code design, potential errors and violations of coding standards, as well as automated code review tasks. In addition, in later stages such as test reporting / error reporting and bug fixing stages, large language models are used to perform tasks such as error analysis, debugging and repair. These tasks are critical at the end of the testing phase when software errors need to be resolved in preparation for product release. These methods test specific code snippets or specific problems, without taking into account the complexity and correlation in the software system and the various quality characteristics tests that are concerned about the system-level safety of safety-critical software. The degree of integration and systematicity are insufficient, and it is difficult to meet the needs of safety-critical software quality assurance in actual industrial scenarios.

[0006] At present, there are some intelligent software evaluation systems that support code interpretation, unit test generation and intelligent debugging (Debug) during the code testing process, mainly for testing single code snippets. However, safety-critical software has unique systematic characteristics, which makes it impossible to systematically and comprehensively evaluate the software under test.

[0007] Therefore, in view of this, the existing structural deficiencies are studied and improved, and a software testing method and system based on a large model is proposed. Summary of the invention

[0008] The purpose of the present invention is to provide a software testing method and system based on a large model to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solution: a software testing method based on a large model, comprising the following steps: Step 1: Describe the code review rules and expand the specifications to be checked: First, list the check specifications that the code needs to meet. According to the combination of thought chain, zero- / few-shot examples, context learning or one or more techniques, describe the check rules in the above list as reasoning instructions of the model according to the check points and check steps. Step 2: Code slicing, structural analysis and dependency analysis of the software project under test: First, the code is sliced ​​according to functions using a code parsing tool. The function fragment information contains the required variable declarations, macro definitions, and context call relationship content of the function. On this basis, the function call graph is obtained through function call relationship analysis, and the control flow graph is obtained through control flow analysis. The function dependency graph is constructed based on the above information, and the program code is divided into code blocks according to the structure information and dependency graph; Step 3: Prompt word construction, combining code review rules and code blocks: First, according to the code review rules, determine the prompt word template configured for the rule in the current round, which includes the role designation of the model, task description, review rule description placeholder, tested code and involved variable declaration placeholder, context information placeholder, output content and format template. Secondly, fill the code rules and inspection steps described in step 1 into the review rule description placeholder. Finally, fill the code block containing the tested code, variable declaration and context information obtained in step 2 into the template placeholder according to the corresponding content. Step 4: Defect inspection, perform defect detection on the code under test: First, the constructed prompt words are input into the target large model in sequence to instruct the large model to find the code quality problems such as violation of specifications, incorrect coding, syntax error, type error, name error, reference error, division by zero error, array out of bounds, and unsigned number overflow based on the code review rules and code fragment information; If the model check result is that there is no defect, go to step 5 for result review; if the model check result is that more context information is needed, go to step 2 to retrieve more context functions according to the program dependency graph, and repeat steps 3-4; if the model returns defects and related content in the required format, go to step 5 for result review; Step 5: Result review, check and confirm the defect report: First, according to the configured prompt word template, the tested code block and the defect inspection report of step 4 are filled in the corresponding positions to complete the result review prompt word construction. Secondly, the result review prompt words are input into the target large model in sequence to instruct the model to complete the review of the defect inspection report according to the prompt word instructions. On this basis, the code inspection report is output in the specified format according to whether there are defects. Finally, the defect reports of all code blocks are aggregated to form a project-level comprehensive software test report.

[0010] Furthermore, the check specifications that the code in step 1 needs to meet are listed including: coding rule checking, runtime defect detection, national military standard C language programming safety subset, and aerospace C language programming specifications.

[0011] Furthermore, a system for the large model-based software testing method includes a project management module, a test pipeline configuration module, an environment containerization configuration and creation module, a test automation execution module, and an evaluation report generation module.

[0012] Furthermore, in the project management module, the user creates a new evaluation project and specifies the ID, version number, and description information of the project, and then uploads the rule set and source code file in the project.

[0013] Furthermore, in the test rule configuration module, the user selects a list of code rules to be reviewed and completes the configuration of the parameters required for the rules.

[0014] Furthermore, in the environment containerization configuration and creation module, the user completes the basic environment, computing resources, and model version for the operation of the evaluation algorithm, and generates an independent container with the corresponding configuration.

[0015] Furthermore, in the test automation execution module, the system converts the preset pipeline into commands to be executed in sequence according to the review rules selected by the user in the evaluation pipeline configuration module and the detailed parameters of each link, and executes each stage and step in the pipeline in the container, and saves all test results in the container storage space.

[0016] Furthermore, the evaluation report generation module generates a final evaluation report based on the test configuration made by the user and the evaluation results saved during the test execution process, wherein the evaluation report includes an overall report and a detailed defect report of violations of various rules.

[0017] The present invention provides a software testing method and system based on a large model, which has the following beneficial effects: the present invention is oriented to the code-level and system-level testing requirements of the software, and the evaluation content includes code specifications, potential logical errors and performance issues, algorithm correctness, etc., which improves the depth and breadth of software testing, and finally forms a comprehensive evaluation report to provide protection for the application of safety-critical software. The present invention also proposes a software intelligent evaluation system including a project management module, a test pipeline configuration module, an environment containerization configuration and creation module, a test automation execution module, and an evaluation report generation module, which has a total of 5 modules, and can implement convenient project management, custom configuration, and highly automated test execution for software evaluation tasks, effectively improving evaluation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a software testing method based on a large model according to the present invention; Figure 2 The present invention is a schematic diagram of a software testing system architecture based on a large model. DETAILED DESCRIPTION

[0019] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0020] like Figure 1-Figure 2 As shown, a software testing method based on a large model includes the following steps: Step 1: Describe the code review rules and expand the specifications to be checked.

[0021] First, list the inspection specifications that the code needs to meet, such as: coding rule inspection, runtime defect detection, national military standard C language programming safety subset, aerospace C language programming specifications, etc. On this basis, according to the combination of one or more technologies such as thinking chain, zero- / few-shot examples, context learning, etc., the inspection rules in the above list are described as the reasoning instructions of the model according to the inspection points and inspection steps.

[0022] Step 2: Code slicing, structural analysis and dependency analysis of the software project under test.

[0023] First, use code parsing tools (such as tree-sitter, joern, etc.) to slice the code according to functions. The function fragment information contains the required variable declarations, macro definitions, function context call relationships, etc., making it easier to query the dependencies between operations. On this basis, the function call graph is obtained through function call relationship analysis, and the control flow graph is obtained through control flow analysis. Then, the function dependency graph is constructed based on the above information, and the program code is divided into code blocks according to the structure information and dependency graph.

[0024] Step 3: Construct prompt words and combine code review rules and code blocks.

[0025] First, according to the code review rules, determine the prompt word template configured for the rule in the current round, which includes the role designation of the model, task description, review rule description placeholder, tested code and involved variable declaration placeholder, context information placeholder, output content and format template. Secondly, fill the code rules and inspection steps described in step 1 into the review rule description placeholder. Finally, fill the code block containing the tested code, variable declaration and context information obtained in step 2 into the template placeholder according to the corresponding content.

[0026] Step 4: Defect inspection, perform defect detection on the code under test.

[0027] First, the constructed prompt words are input into the target large model in sequence to indicate the code review rules and code fragment information that the large model is based on, and to discover code quality problems such as violations of specifications, incorrect encoding, syntax errors, type errors, name errors, reference errors, division by zero errors, array out of bounds, and unsigned number overflows.

[0028] If the model check result is that there are no defects, go to step 5 to review the results; if the model check result is that more context information is needed, go to step 2 to retrieve more context functions according to the program dependency graph, and repeat steps 3-4; if the model returns defects and related content in the required format, go to step 5 to review the results.

[0029] Step 5: Review the results, check and confirm the defect report.

[0030] First, according to the configured prompt word template, the tested code block and the defect inspection report of step 4 are filled in the corresponding positions to complete the result review prompt word construction. Secondly, the result review prompt words are input into the target large model in sequence to instruct the model to complete the review of the defect inspection report according to the prompt word instructions. On this basis, the code inspection report is output in the specified format according to whether there are defects. Finally, the defect reports of all code blocks are aggregated to form a project-level comprehensive software test report.

[0031] A software testing system based on a large model provided in an embodiment of the present invention includes a project management module, a test pipeline configuration module, an environment containerization configuration and creation module, a test automation execution module and an evaluation report generation module.

[0032] In the project management module, users create a new evaluation project and specify the project's ID, version number, description and other information, and then upload the rule set, source code and other files in the project; In the test rule configuration module, the user selects the code rule list to be reviewed and completes the configuration of the parameters required for the rule; In the environment containerization configuration and creation module, the user completes the basic environment, computing resources, and model version for the evaluation algorithm to run, and generates an independent container with the corresponding configuration; In the test automation execution module, the system converts the preset pipeline into commands to be executed in sequence according to the review rules and detailed parameters of each link selected by the user in the evaluation pipeline configuration module, and executes each stage and step in the pipeline in the container. Furthermore, all test results are saved in the container storage space; The evaluation report generation module generates a final evaluation report based on the test configuration made by the user and the evaluation results saved during the test execution. The evaluation report includes an overall report and a detailed defect report that violates various rules, such as the variables, code lines, preconditions, etc. where the defects occur.

[0033] The present invention includes code review rule description, code slicing, prompt word construction, defect inspection, and result review: 1) Code review rule description, which is used to describe the standardization, correctness, performance indicators and other information that the tested code needs to meet, which should also include the inspection example information of the corresponding rules; 2) Code slicing is used to parse the code of the project under test, including the variable declaration, parameter type, context, call dependency and other structural information of the function, determine the control flow and dependency graph information of the code to be tested from the structural information, and parse the code into code blocks; 3) The prompt word construction stage is used to combine the code snippet information and the code review rule description to indicate the code review rules and code snippet information based on the large model, and generate a code review report corresponding to the tested code.

[0034] The prompt word construction process includes: obtaining the prompt word template for the current round configuration, including model role specification, task description, review rule description placeholder, tested code and involved variable declaration placeholder, context information placeholder, output content and format template; 4) Defect inspection phase, which is used to send the prompt words constructed based on the code rules and the tested code in step 3) into the large model for defect inspection to find code quality problems such as violation of specifications, incorrect coding, syntax errors, type errors, name errors, reference errors, division by zero errors, array out of bounds, unsigned number overflow, etc.

[0035] When the output of the large model is that there is insufficient information to confirm whether the code has defects and more information is needed for defect inspection, steps 2 to 4 are repeated to obtain more context information and perform defect inspection until a definite result is returned; 5) The result confirmation phase is used to instruct the large model to review the defect inspection results to confirm their correctness, including sending the tested code information and defect inspection results to the large language model to instruct the model to perform secondary confirmation of the inspection results.

[0036] In order to solve the above technical problems, the present invention also provides a software intelligent evaluation system based on a large model, including a project management module, a test rule configuration module, an environment containerization configuration and creation module, a test automation execution module and an evaluation report generation module.

[0037] 1) The project management module is used to create, delete, and configure basic information for assessment projects, and can upload or remove review rule files and code files. In addition, it can also display the progress of assessment activity execution ("to be executed", "in progress", "completed"); 2) In the test rule configuration module, users can customize the various code rules to be reviewed and configure the parameters required for each rule; 3) In the environment containerization configuration and creation module, the user specifies the basic environment, file dependencies, software dependency resources and other information for this evaluation run, and the system generates an independent container with the corresponding configuration to run the various tests configured in the test pipeline configuration.

[0038] 4) The test automation execution module is used to convert the user-configured evaluation pipeline and corresponding parameters into sequentially arranged commands. These commands are executed in sequence in the container. All input data, intermediate data, output data and evaluation results are in the storage space specified by the container.

[0039] 5) The evaluation report is used to conduct a comprehensive analysis of the results of the above test links, present the test results in the form of charts, and point out potential software defects in the tested code.

[0040] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A software testing method based on a large model, characterized in that: The following steps are involved: Step 1: Describe the code review rules and expand the specifications to be checked: First, list the check specifications that the code needs to meet. According to the combination of thought chain, zero- / few-shot examples, context learning or one or more techniques, describe the check rules in the above list as reasoning instructions of the model according to the check points and check steps. Step 2: Code slicing, structural analysis and dependency analysis of the software project under test: First, the code is sliced ​​according to functions using a code parsing tool. The function fragment information contains the required variable declarations, macro definitions, and context call relationship content of the function. On this basis, the function call graph is obtained through function call relationship analysis, and the control flow graph is obtained through control flow analysis. The function dependency graph is constructed based on the above information, and the program code is divided into code blocks according to the structure information and dependency graph; Step 3: Prompt word construction, combining code review rules and code blocks: First, according to the code review rules, determine the prompt word template configured for the rule in the current round, which includes the role designation of the model, task description, review rule description placeholder, tested code and involved variable declaration placeholder, context information placeholder, output content and format template. Secondly, fill the code rules and inspection steps described in step 1 into the review rule description placeholder. Finally, fill the code block containing the tested code, variable declaration and context information obtained in step 2 into the template placeholder according to the corresponding content. Step 4: Defect inspection, perform defect detection on the code under test: First, the constructed prompt words are input into the target large model in sequence to instruct the large model to find the code quality problems such as violation of specifications, incorrect coding, syntax error, type error, name error, reference error, division by zero error, array out of bounds, and unsigned number overflow based on the code review rules and code fragment information; If the model check result is that there is no defect, go to step 5 for result review; if the model check result is that more context information is needed, go to step 2 to retrieve more context functions according to the program dependency graph, and repeat steps 3-4; if the model returns defects and related content in the required format, go to step 5 for result review; Step 5: Result review, check and confirm the defect report: First, according to the configured prompt word template, the tested code block and the defect inspection report of step 4 are filled in the corresponding positions to complete the result review prompt word construction. Secondly, the result review prompt words are input into the target large model in sequence to instruct the model to complete the review of the defect inspection report according to the prompt word instructions. On this basis, the code inspection report is output in the specified format according to whether there are defects. Finally, the defect reports of all code blocks are aggregated to form a project-level comprehensive software test report.

2. A software testing method based on a large model according to claim 1, characterized in that: The list of inspection specifications that the code needs to meet in step 1 includes: coding rule checking, runtime defect detection, national military standard C language programming safety subset, and aerospace C language programming specifications.

3. A system for a software testing method based on a large model as claimed in claim 2, characterized in that: It includes project management module, test pipeline configuration module, environment containerization configuration and creation module, test automation execution module and evaluation report generation module.

4. The system of a software testing method based on a large model according to claim 3, characterized in that: In the project management module, the user creates a new evaluation project and specifies the project's ID, version number, and description information, and then uploads the rule set and source code files in the project.

5. The system of a software testing method based on a large model according to claim 4, characterized in that: In the test rule configuration module, the user selects the list of code rules to be reviewed and completes the configuration of the parameters required for the rules.

6. The system of a software testing method based on a large model according to claim 5, characterized in that: In the environment containerization configuration and creation module, the user completes the basic environment, computing resources, and model version for the running of the evaluation algorithm, and generates an independent container with the corresponding configuration.

7. The system of a software testing method based on a large model according to claim 6, characterized in that: In the test automation execution module, the system converts the preset pipeline into commands to be executed in sequence according to the review rules selected by the user in the evaluation pipeline configuration module and the detailed parameters of each link, and executes each stage and step in the pipeline in the container, and saves all test results in the container storage space.

8. The system of the software testing method based on a large model according to claim 7, characterized in that: The evaluation report generation module generates a final evaluation report based on the test configuration made by the user and the evaluation results saved during the test execution. The evaluation report includes an overall report and a detailed defect report that violates various rules.

Images