A Method for Automated Software Testing Execution and Result Analysis Based on Generative Large Models

Abstract

This invention discloses a method for automated software testing execution and result analysis based on a generative large model, belonging to the field of software testing technology. The method includes constructing an initial test knowledge base, generating a structured test knowledge graph through information extraction and knowledge fusion; extracting test target descriptions, combining them with a preset test depth and the current version of the tested object as input to the generative large model, outputting a preliminary script set containing natural language test steps and expected assertions, converting it into executable scripts, and encoding verification rules. After executing the scripts, actual output and resource consumption records are collected, compared to generate a discrepancy report, and analyzed by a defect pattern recognition model to generate a defect suspicion report. This method reduces manual maintenance costs and improves the accuracy of complex scenario coverage by dynamically generating scripts using knowledge graphs and large models, and automatically identifies defects behind discrepancies using defect pattern models to shorten the analysis cycle and reduce the possibility of missed detections.

Description

Technical Field

[0001] This invention belongs to the field of software testing technology, specifically a method for automated software testing execution and result analysis based on generative large models. Background Technology

[0002] Conventional automated software testing execution and result analysis methods largely rely on predefined templates or recorded operations to generate test scripts. Result analysis is then performed by manually verifying the actual output against the expected assertions. Template-based methods require design for fixed scenarios and are difficult to adapt to frequent software version changes and complex testing objectives; recording methods are limited by operation paths and cannot cover logic branches that have not been executed. In result analysis, manually comparing outputs one by one is prone to missing hidden deviations, and the identification of defect patterns such as performance fluctuations and logical anomalies relies on experience, resulting in low efficiency.

[0003] Existing technical solutions have shortcomings. In the script generation stage, static templates and recording technologies lack dynamic knowledge organization capabilities, requiring extensive manual script reconstruction for new version features or complex business flows, resulting in high maintenance costs. In the results analysis stage, simple assertion matching can only detect explicit discrepancies and cannot automatically identify the types of defects associated with the differences, requiring manual tracing of logs and reproduction of scenarios, which is time-consuming and prone to misjudgment.

[0004] To address the aforementioned problems, this invention constructs an initial test knowledge base and generates a structured test knowledge graph through information extraction and knowledge fusion. From this graph, test target descriptions are extracted. These descriptions, combined with a preset test depth and the current software version information of the tested object, are input into a generative large model. The model outputs a preliminary set of test scripts containing natural language test steps and expected result assertion descriptions. These scripts are then transformed into machine-executable automated test scripts, and the assertion descriptions are logically encoded to generate verifiable rules. Simultaneously, after executing the scripts in the target environment, actual output data and system resource consumption records are collected and compared with the assertion rules to generate a preliminary discrepancy report. This report is then analyzed using a defect pattern recognition model to generate a defect suspicion analysis report. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art; To this end, this invention proposes a method for automated software testing execution and result analysis based on generative large models, including: Construct an original test knowledge base, extract information and fuse knowledge from the original test knowledge base, and generate a structured test knowledge graph; The test target description is extracted from the structured test knowledge graph, and the test target description, the preset test depth and the current software version information of the tested object are input into the generative large model. The output is a preliminary test script set including natural language test steps and expected result assertion descriptions. The natural language testing steps are converted into executable versions to generate machine-executable automated test scripts. At the same time, the expected result assertion descriptions are logically encoded to generate verifiable assertion verification rules. Deploy and execute the automated test script in the target test environment, collect the actual output data and system resource consumption records generated during the execution process, and form the original test execution record; The original test execution records are compared with the assertion verification rules to generate a preliminary result difference report. The preliminary result difference report is then analyzed using a defect pattern recognition model to generate a defect suspicion analysis report.

[0006] Furthermore, information extraction and knowledge fusion are performed on the original test knowledge base to generate a structured test knowledge graph, including: Collect test task descriptions, product requirement documents, and historical test cases of the software under test to build an original test knowledge base; The structured test knowledge graph includes functional entities, test scenario entities, input / output constraint entities, and defect pattern entities, as well as their relationships. The test task description, product requirement document and historical test cases are parsed to identify text fragments related to functions, operations, data constraints and exception handling. Named entity recognition technology is applied to the identified text fragments to label the names of functional components, user behavior phrases, input data value ranges, and expected system response descriptions. Based on the preset software testing domain ontology, the labeled functional component names, user behavior phrases, input data value ranges and expected system response descriptions are mapped to instances of the functional entities, test scenario entities, input / output constraint entities and defect mode entities; Based on the software business process logic and the step logic of test cases, call, containment, pre-requirement, post-requirement and equivalence class relationships are established among the identified entity instances to complete knowledge fusion and construct the structured test knowledge graph.

[0007] Furthermore, the natural language testing steps are converted into executable versions to generate machine-executable automated test scripts, including: Select the appropriate automated testing framework for the application type to which the object under test belongs; The core actions, operation objects, and input data in the natural language testing steps are analyzed. The core actions are mapped to the standard operation functions of the automated testing framework, the operation objects are located as element identifiers supported by the automated testing framework, and the input data is encapsulated into data parameters that meet the requirements of the function interface. Based on the order of the natural language testing steps described, the mapped standard operation functions, element identifiers, and data parameters are arranged and combined, and necessary synchronization wait and exception handling control logic is inserted to compile and generate the automated test script that can be executed by the test engine.

[0008] Furthermore, the expected result assertion description is simultaneously logically encoded to generate verifiable assertion verification rules, including: From the description of the expected result assertion, the expected output result to be verified, the verification condition, and the verification logic are separated. The expected output result is transformed into a specific expected value, which can be text, numerical value, Boolean value or complex data structure; The verification conditions are translated into comparison operators, including equal to, contain, greater than, and between; By combining the verification logic with expected values ​​and comparison operators, and using the syntax structure of the assertion programming framework, the assertion verification rules that can automatically verify the actual output data at runtime are constructed.

[0009] Furthermore, the original test execution records are compared with the assertion verification rules to generate a preliminary result difference report, including: Under the control of the test execution agent, the actual output data that needs to be verified at the assertion time is extracted from the original test execution record; According to the execution order of the assertion verification rules, the actual output data is matched and calculated with the expected value and comparison operator defined in the assertion verification rules in turn; When the comparison result between the actual output data and the expected value does not meet the conditions defined by the comparison operator, the assertion verification failure is recorded, and the time of failure, the test steps involved, the actual output data, the expected value, and the comparison operator used are recorded as a difference record. All discrepancies generated during the execution of a complete test script are summarized and accompanied by execution environment information to form a structured preliminary result discrepancy report.

[0010] Furthermore, the analysis of the preliminary result difference report using the defect pattern recognition model to generate a defect suspicion analysis report includes: Based on a historical defect database, the defect pattern recognition model is trained, and the defect pattern recognition model can identify the difference features that match known defect patterns; Extract the difference pattern features from the preliminary result difference report. The difference pattern features include the position of the difference in the test process, the data types involved, the keywords of the error message, and the difference chain. The difference pattern features are input into the trained defect pattern recognition model, and the defect pattern recognition model outputs the matching probability of each difference record with each known defect pattern. For each discrepancy record, the defect pattern with the highest matching probability is selected as the candidate defect cause. All discrepancy records are sorted according to the matching probability, and the sorting results are associated with the candidate defect causes to generate the defect suspicion analysis report.

[0011] Furthermore, it also includes: inputting the preliminary result difference report, the defect suspicion analysis report, and the system resource consumption record in the original test execution record into the test result comprehensive analysis model to generate a comprehensive test analysis report; The comprehensive test analysis report, the original test knowledge base, and all data collected in the current test round are input into the generative large model. The generative large model then outputs an iterative test optimization scheme that includes test coverage assessment, test script optimization suggestions, and subsequent test task planning. The generation of the comprehensive test analysis report specifically includes: The test result comprehensive analysis model receives the preliminary result difference report, defect suspicion analysis report and system resource consumption record as input; The comprehensive analysis model of the test results contains a multi-dimensional correlation analysis network, which simultaneously analyzes differences at the functional level, root causes of defects, and anomalies at the resource level, and seeks correlation patterns among these differences, root causes of defects, and anomalies at the resource level. When a strong temporal correlation is detected between a specific functional difference, a specific candidate defect cause, and a specific abnormal pattern of system resource consumption, the multi-dimensional correlation analysis network marks this situation as a high-confidence defect correlation event. Summarize all the difference analysis results and defect suspicion analysis results, and integrate the high-confidence defect association events to generate a comprehensive test analysis report that includes test conclusions, defect risk distribution, and performance bottleneck indications.

[0012] Furthermore, it also includes: inputting the comprehensive test analysis report, the original test knowledge base, and all data collected in the current test round into the generative large model, and having the generative large model output an iterative test optimization scheme that includes test coverage assessment, test script optimization suggestions, and subsequent test task planning, specifically including: The comprehensive test analysis report, the original test knowledge base, the automated test scripts executed in the current test round, the original test execution records, the preliminary result difference report, and the defect suspicion analysis report are integrated and organized into structured multi-round test context information; The multi-round test context information and the background information of software iterative development are encapsulated together to form test optimization prompt information for generative large models; The test optimization prompts are sent to the generative big model, which is then instructed to focus on evaluating existing coverage, identifying missing scenarios, correcting existing script defects, and planning subsequent tests. The system receives the response text from the generative large model regarding the test optimization prompts, performs structured parsing on the response text, extracts information on test coverage assessment, suggestions for modifying existing automated test scripts, and subsequent test task planning for uncovered functions or scenarios, and organizes the extracted content into the iterative test optimization scheme.

[0013] Furthermore, the structured parsing of the response text to extract information on test coverage assessment, suggestions for modifying existing automated test scripts, and subsequent test task planning for uncovered functions or scenarios includes: In the response text, locate the text paragraph that begins with an assessment of the existing test coverage, parse the enumeration list of covered functional points, boundary conditions, and abnormal scenarios in the text paragraph, identify the obvious omissions, and organize the above information into the test coverage assessment section. In the response text, locate the text paragraph that begins with "Optimize existing test scripts", parse the descriptions in the text paragraphs that indicate logical errors, invalid locators, and inaccurate assertion conditions in the automated test scripts, and extract the correction suggestions for each description and organize them into the modification suggestion section for the automated test scripts. In the response text, locate the text paragraphs that begin with a suggestion to add new test tasks, parse the new test scenarios, new data combinations or new business processes proposed in the text paragraphs, and transform each suggestion into a test task item containing test objectives and a brief description of steps, and organize it into the subsequent test task planning section.

[0014] Furthermore, the process of transforming each proposal into a test task entry that includes a test objective and a brief description of the steps involved includes: Analyze the core verification intent of each proposal in the suggested new test task paragraph, and extract the core verification intent into a test objective statement; Deconstruct the implicit or explicit user action sequences, system interactions, and data changes in each proposal, and organize them into a continuous series of steps from test preparation to result verification; Assign a step number to each step and ensure logical coherence between steps to form a structured and concise step description; Pair the test objective statement with a brief step description to create a complete test task entry, which will guide the generation of the next round of test scripts or manual test design.

[0015] Compared with the prior art, the beneficial effects of the present invention are: An initial test knowledge base is constructed, extracting entities and relationships such as test objects, operation steps, and expected results through information extraction technology. Conflicts and redundancies are then integrated using knowledge fusion technology to generate a structured test knowledge graph. Test target descriptions are extracted from this graph, and combined with a preset test depth and the current software version information of the tested object, input into a generative large model. This outputs a preliminary set of test scripts containing natural language test steps and expected result assertion descriptions, which are then transformed into machine-executable automated test scripts. The assertion description logic is logically encoded to generate verifiable rules. This approach overcomes the static limitations of templates and recordings, dynamically organizing test knowledge through a knowledge graph. Combined with the semantic generation capabilities of the large model, test scripts adaptively adjust to version changes and test targets, reducing manual writing and maintenance costs and improving the accuracy of coverage for complex business scenarios.

[0016] Automated test scripts are deployed and executed in the target test environment. Actual output data and system resource consumption records are collected to form raw test execution records, which are then compared with assertion verification rules to generate a preliminary result difference report. A defect pattern recognition model is used to analyze the deviation characteristics in the report, matching them with common features in a predefined defect pattern library to generate an analysis report annotating suspected defect types, affected modules, and confidence levels. This method replaces manual item-by-item verification, automatically identifying potential defect patterns behind the differences, shortening the analysis cycle, and reducing the possibility of missed detections due to experience limitations. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of the software automated testing execution and result analysis method based on generative large models described in this invention. Figure 2 A flowchart for generating a structured test knowledge graph; Figure 3 A flowchart for generating verifiable assertion validation rules; Figure 4 Resource consumption analysis chart for payment interface testing; Figure 5 A heatmap showing the association between test cases and defect types. Detailed Implementation

[0018] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] See Figure 1A raw test knowledge base is constructed, and information extraction and knowledge fusion are performed on this base to generate a structured test knowledge graph. Test target descriptions are extracted from this structured knowledge graph, and these descriptions, along with the preset test depth and the current software version information of the tested object, are input into a generative large-scale model. This model outputs a preliminary set of test scripts containing natural language test steps and expected result assertion descriptions. Next, the natural language test steps are converted into executable versions to generate machine-executable automated test scripts. Simultaneously, the expected result assertion descriptions are logically encoded to generate verifiable assertion verification rules. The generated automated test scripts are then deployed and executed in the target test environment, collecting actual output data and system resource consumption records to form raw test execution records. Finally, the raw test execution records are compared with the assertion verification rules to generate a preliminary result difference report. This report is then analyzed using a defect pattern recognition model to generate a defect suspicion analysis report.

[0020] See Figure 2 In one embodiment of the present invention, the software automation testing method based on generative large models involves constructing a structured test knowledge graph. Taking the login function module as an example, the original test knowledge base includes a product requirements document, which states that "users should be able to log in to the system by entering a username and password. If successful, they will enter the homepage; if they fail, they should be prompted with 'username or password incorrect'", a test task description requiring "verifying the positive and abnormal scenarios of the login function", and a historical test case "Test Case ID: L001, Steps: 1. Enter username 'admin', 2. Enter password '123456', 3. Click the login button, expected result: redirect to the homepage dashboard". The structured test knowledge graph includes functional entities, test scenario entities, input / output constraint entities, and defect pattern entities and their relationships. For the login function, the functional entities are instantiated as "login page" and "homepage", and the test scenario entities are instantiated as "successful login scenario" and "login failure scenario".

[0021] Text parsing is performed on product requirement documents, test task descriptions, and historical test cases to identify text fragments related to functions, operations, data constraints, and exception handling. For example, fragments such as "Enter username and password," "Enter homepage," and "Prompt 'Incorrect username or password'" are identified from the requirement document; and operation and result fragments such as "Enter username 'admin'," "Click the login button," and "Redirect to the homepage dashboard" are identified from historical test cases. Named entity recognition technology is applied to the identified text fragments to label functional component names such as "login button," user behavior phrases such as "Enter username" and "Click," input data value ranges such as "username 'admin'," and expected system response descriptions such as "Redirect to the homepage dashboard" and "Prompt 'Incorrect username or password'."

[0022] Based on a pre-defined software testing domain ontology, the labeled information is mapped to entity instances in a structured test knowledge graph. Functional component names "Login Button" and "Homepage" are mapped to instances of functional entities. User behavior phrases "Enter Username" and "Click Login" are mapped to components of the operation sequence under the test scenario entity "Successful Login Scenario." The input data value domain "Username 'admin'" and the expected system response description "Jump to Homepage dashboard" are mapped to instances of input / output constraint entities. Based on the software business process logic and test case step logic, relationships are established between the identified entity instances. An "inclusion" relationship is established between the "Login Page" functional entity and the "Successful Login Scenario" test scenario entity; a "preceding" relationship is established between the "Enter Username" operation and the "Username 'admin'" input constraint; and a "postceding" relationship is established between the "Click Login" operation and the "Jump to Homepage dashboard" output constraint. This completes knowledge fusion and constructs a structured test knowledge graph for the login function module. In some embodiments, the strength of relationships between entities in a structured testing knowledge graph can be quantified by a weighting coefficient. This weighting coefficient reflects the frequency or importance of the relationship in historical tests. One possible calculation method is that the weighting coefficient equals the number of times the relationship is referenced in historical test cases divided by the total number of times all relationships are referenced, i.e.: in: The weight coefficients representing a specific relationship. This indicates the number of times that specific relationship has been referenced in all historical test cases. This represents the total number of times all relations in the structured test knowledge graph are referenced.

[0023] In practical implementation, the original test knowledge base can also include user manuals and interface documents. By uniformly parsing and extracting information from these multi-source heterogeneous documents, the attributes and relationships of entities in the structured test knowledge graph can be further enriched. Optionally, named entity recognition technology can combine rule-based methods and pre-trained model-based methods. First, a predefined domain dictionary and pattern rules are used for preliminary matching and annotation. Then, a sequence labeling model finely tuned on a software engineering corpus is used to perform deep semantic understanding and entity recognition on unmatched text fragments to improve the accuracy and coverage of annotation. In some embodiments, the software testing domain ontology defines the core attributes and inheritance system of functional entities, test scenario entities, input / output constraint entities, and defect pattern entities. For example, functional entities can have attributes such as "component name," "component type," and "module to which they belong." Input / output constraint entities can be further distinguished into classes such as "preconditions," "input data," "expected output," and "postconditions." Mapping based on the domain ontology ensures that knowledge extracted from different document sources has a consistent semantic representation. When establishing relationships based on software business process logic, optionally, business process modeling symbols can be used to model key user paths. From the model, state transition sequences and operation sequences of business objects can be extracted, and these sequences can be transformed into "call" and "post" relationship chains between entities in a structured test knowledge graph. This allows the generated knowledge graph to more completely reflect the business logic. It can be understood that the construction of a structured test knowledge graph is an iterative process. New test cases and defect reports generated in subsequent test rounds can be continuously incorporated into the original test knowledge base. Through incremental information extraction and knowledge fusion, the content of the structured test knowledge graph is dynamically updated and expanded, keeping it synchronized with the evolution of the software under test.

[0024] See Figure 3In one embodiment of the present invention, the software automation testing method based on generative large models involves converting a generated preliminary test script set into an executable form. Assuming this is for the "Add to Cart" function of an e-commerce website, a natural language test step output by the generative large model is described as "On the product list page, click the 'Add to Cart' button for the first product," and the expected result assertion is described as "The page should display a pop-up message with the text 'Added successfully.'" The corresponding automated testing framework, SeleniumWebDriver, is selected for the web application type to which the tested object belongs. SeleniumWebDriver provides a series of standard operation functions and element location mechanisms for simulating browser operations. The natural language test step "On the product list page, click the 'Add to Cart' button for the first product" is parsed, identifying the core action as "click," the operation object as "the 'Add to Cart' button for the first product," and the input data in this step is empty. The core action "click" is mapped to the standard operation function `click()` of the Selenium WebDriver automated testing framework. The operation object, "the 'Add to Cart' button for the first item," is located as an element identifier supported by the Selenium WebDriver framework by analyzing the page's document object model structure. Following the order of the natural language test steps, the mapped standard operation function `click()` is combined with the element identifier, and necessary synchronous wait logic is inserted to ensure the element is clickable.

[0025] In practical implementation, the expected result assertion description "the page should pop up a prompt box, and the prompt box text contains 'added successfully'" is logically encoded. The expected output result to be verified is extracted from the expected result assertion description as "prompt box text," the verification condition is "contains 'added successfully'," and the verification logic is "AND" (representing logical AND). The expected output result is transformed into a specific expected value, i.e., the string "'added successfully'". The verification condition "contains" is translated into a comparison operator, typically corresponding to the `contains` or `in` operator in programming context. The verification logic, expected value, and comparison operator are combined, and using the syntax structure of an assertion programming framework such as PyTest, assertion verification rules are constructed that can automatically verify the actual output data at runtime. For this description, the generated assertion verification rule might be: `assert "added successfully" inert_text`, where `alert_text` is the actual pop-up text variable captured from the execution environment. In some embodiments, the selection of the automated testing framework depends on the application's technology stack. For mobile applications, the Appium framework can be selected; for interface testing, the Requests library combined with PyTest can be selected. The purpose of selecting the framework is to obtain a standard function library for interacting with the object under test. When parsing natural language testing steps, optionally, dependency parsing techniques can be used to identify the predicate of the action, the object of the action, and the modifiers of the object, thereby more accurately separating the core action, the operation object, and the input data. For example, from "enter the keyword 'laptop' into the search box," the action "input," the object "search box," and the data "'laptop'" can be parsed. Locating the operation object as an element identifier supported by the automated testing framework is a key step. In some embodiments, the robustness of the location can be improved by combining multiple location strategies, such as using element IDs, XPath, and CSS selectors simultaneously, and implementing a priority attempt logic in the script. The probability function of successful location can be expressed as: in: This indicates the probability of successful final location. Indicates the number of alternative positioning strategies. Indicates the first The applicability coefficient of a positioning strategy in the current page context (value range from 0 to 1). Indicates the first The historical success rate of the positioning strategy (range 0 to 1). It's understandable that inserting necessary synchronous wait and exception handling control logic is to handle the dynamic loading and network latency issues of web applications. Synchronous waits typically use explicit wait conditions, while exception handling uses try-except blocks to wrap potentially failing operations to ensure script robustness.

[0026] In practical implementation, the logical encoding of the expected result assertion description needs to handle more complex assertion structures. For example, for the assertion description "the response status code should be 200, and the 'stock' field value in the response body is greater than 0", the separated expected output results include "response status code" and "the 'stock' field value in the response body", with verification conditions of "should be 200" and "greater than 0", and the verification logic is "AND". The expected output results are converted into expressions for the numerical expected value 200 and the numerical expected value greater than 0, respectively, and the verification conditions are translated into comparison operators equal to and greater than. Using the syntax structure of the assertion programming framework, these elements are combined to generate a verifiable assertion verification rule: assertresponse.status_code==200andresponse.json()['stock']>0. Optionally, when separating the verification component from the expected result assertion description, rule templates or finite state machines can be used to parse common assertion expression patterns, such as "should be equal to X", "should contain Y", and "should be between A and B". These patterns can be directly mapped to specific matchers in the assertion library. It is understandable that the process of combining verification logic with expected values ​​and comparison operators to construct assertion verification rules is essentially generating a piece of code that can automatically perform boolean judgments during test runtime. The triggering of this code depends on the callback mechanism when the automated test script reaches a specific checkpoint.

[0027] In one embodiment of the present invention, the software automation testing method based on generative large models involves generating difference reports and defect analysis. Taking the testing of a file upload function module as an example, an automated test script is executed in the target test environment and the original test execution record is collected. The original test execution record includes the test step execution sequence, the actual output data of each step, and a snapshot of system resource consumption. Under the control of the test execution agent, the actual output data that needs to be verified at the assertion time is extracted from the original test execution record. For an assertion that verifies "the page displays a green success message after successful upload", the extracted actual output data may be the captured page prompt message text, such as "file upload failed". According to the execution order of the assertion verification rules, the actual output data is matched and calculated with the expected value and comparison operator defined in the assertion verification rules. In this example, the assertion verification rule is defined as assert "file upload successful" inactual_message, where the expected value is the string "file upload successful" and the comparison operator is in (includes). The actual output data "file upload failed" is compared with the expected value "file upload successful" to determine whether "file upload failed" contains "file upload successful". The calculation result is that the condition is not met. When the comparison between the actual output data and the expected value does not meet the conditions defined by the comparison operator, an assertion verification failure is recorded. The time of failure, the involved test steps, the actual output data "file upload failed," the expected value "file upload successful," and the comparison operator "in" used are recorded as a difference record. All difference records generated during the execution of a complete test script are summarized. For example, in addition to the upload result assertion failure, there may be difference records for steps such as file size verification and file type verification. These difference records, along with test environment information such as browser version and execution timestamp, are integrated to form a structured preliminary result difference report.

[0028] The defect pattern recognition model is trained on a historical defect database, which contains various software defects recorded in past projects, along with their corresponding error phenomena, stack traces, and fixes. The model can identify discrepancy features that match known defect patterns. Discrepancy pattern features are extracted from the preliminary result discrepancy report. For a file upload failure discrepancy record, the extracted features include the discrepancy's position in the test process ("file upload result verification stage"), the data type involved ("string-type prompt message"), the keyword "failure" in the error message, and possible discrepancy chains. For example, in this test, the "file selection" step before the file upload step also recorded a warning log, forming a potential discrepancy chain. The extracted discrepancy pattern features are input into the trained defect pattern recognition model. Internally, the model vectorizes the features and calculates similarity with the stored known defect pattern feature library. The model outputs the matching probability of each discrepancy record with each known defect pattern. For example, the current discrepancy record has a matching probability of 0.85 with the "backend file size limit defect" pattern and 0.60 with the "frontend file type verification defect" pattern. For each discrepancy record, select the defect pattern with the highest matching probability as the candidate defect cause. Sort all discrepancy records according to the matching probability. For example, sort the discrepancy record with a matching probability of 0.85 before the discrepancy record with a matching probability of 0.60. Associate the sorting results with the candidate defect cause "backend file size limit defect" to generate a defect suspicion analysis report. The report lists the discrepancies in descending order of suspicion and their most likely causes.

[0029] In some embodiments, the process of extracting difference pattern features from the preliminary result difference report can be more refined. The position of the difference in the testing process can be encoded as a step path hash value from the test case start point to the assertion failure point. The data types involved can be subdivided into basic types, set types, or custom object types. The keywords in the error message can be formed into a bag-of-features after removing stop words and stemming. The defect pattern recognition model can adopt a neural network architecture based on an attention mechanism. This model calculates the cosine similarity between the difference feature vector and each known defect pattern vector as the basis for the matching probability, and introduces a correction term based on historical co-occurrence frequency. The final matching probability calculation can be expressed as: in: This represents the final matching probability score. This represents the difference feature vector of the current difference record. The feature vector representing a known defect pattern. This indicates the calculation of cosine similarity. This indicates the number of times the difference feature and the defect pattern co-occur in historical data. This is a weighting coefficient between 0 and 1, used to balance semantic similarity and statistical co-occurrence information. It's understood that training the defect pattern recognition model relies on a high-quality historical defect database, which needs continuous maintenance and updates to incorporate newly identified defects and their patterns, enabling the model to adapt to new defect types emerging during software project iterations. Optionally, defect patterns can include not only functional defects but also performance defect patterns, such as "slow memory growth" or "stepwise increase in response time." Corresponding difference pattern features need to be extracted from system resource consumption records, such as the morphological characteristics of time-series data like memory usage curves and CPU utilization curves. In some embodiments, the construction of difference chains is achieved by analyzing the correlation between timestamps and event types in log events in the original test execution records. Multiple difference records that are temporally close and logically causally related are linked together to form a difference chain, which serves as richer difference pattern features input to the defect pattern recognition model. It's understood that the goal of generating a defect suspicion analysis report is to provide testers with priority guidance for problem investigation. The ranking of matching probabilities reflects the likelihood that the difference is caused by a specific known defect pattern, but the final confirmation of the defect requires manual activities such as code review or debugging.

[0030] In one embodiment of the present invention, the software automation testing method based on a generative large model involves comprehensive analysis and iterative optimization of test results. Considering a test round of an online payment gateway interface, this round generates a preliminary result difference report, a defect suspicion analysis report, and original test execution records containing system resource consumption records. The preliminary result difference report, defect suspicion analysis report, and system resource consumption records are input into a comprehensive test result analysis model. After receiving these inputs, the multi-dimensional correlation analysis network contained within the model begins to operate. The multi-dimensional correlation analysis network simultaneously analyzes functional-level differences, defect root causes, and resource-level anomalies, seeking correlation patterns between these factors. For example, a functional difference might manifest as "the payment interface returns a 'system busy' error," the defect suspicion analysis report might suggest "database connection pool exhaustion," and the resource consumption record might show "database connection count approaching maximum limit and response time surging" near the time the error occurred. When a strong temporal correlation is detected between a specific functional difference, a specific candidate defect cause, and a specific system resource consumption anomaly pattern, the multi-dimensional correlation analysis network marks this situation as a high-confidence defect correlation event. By summarizing all the difference analysis results and defect suspicion analysis results, and integrating the marked high-confidence defect-related events, a comprehensive test analysis report is generated, which includes test conclusions, defect risk distribution, and performance bottleneck indications. The report will clearly point out that "the payment failure problem is highly likely to be caused by database connection resource bottlenecks."

[0031] In practice, the comprehensive test analysis report, the original test knowledge base, and all data collected in the current test round are input into the generative big data model to obtain optimization solutions. First, the comprehensive test analysis report, the original test knowledge base, the automated test scripts executed in the current test round, the original test execution records, the preliminary result difference report, and the defect suspicion analysis report are integrated and organized into structured multi-round test context information. This multi-round test context information exists in the form of a summary of the previous round's test execution, a problem summary, and the current system status. This multi-round test context information is then encapsulated with background information on software iteration development, such as the list of new features added in this iteration and the scope of modified modules, to form test optimization prompts for the generative big data model. These prompts are sent to the generative big data model, explicitly specifying that the model should focus on evaluating existing coverage, identifying missing scenarios, correcting existing script defects, and planning subsequent tests. Finally, the generative big data model receives response text to the test optimization prompts; this response text consists of one or more paragraphs of natural language descriptions of evaluation and suggestions. The response text is structured and parsed to extract information on test coverage assessment, suggestions for modifying existing automated test scripts, and plans for subsequent test tasks for uncovered functions or scenarios. The extracted content is then organized into a clear iterative test optimization plan to guide the testing team's next actions.

[0032] In some embodiments, the multi-dimensional association analysis network calculates association strength based on the overlap of events within a time window and the support of historical patterns. One quantitative method for calculating association strength is as follows: in: This indicates the strength of the association between functional differences, candidate defect causes, and abnormal resource consumption. This indicates the duration of overlap between three events on the timeline. This indicates the total observation time window used in the analysis. This indicates the matching probability score given in the defect suspicion analysis report for this discrepancy. This represents a normalized score indicating the degree of deviation of abnormal resource consumption from the baseline. When the correlation strength... When the threshold is exceeded, the group of events is marked as a high-confidence defect association event. It can be understood that the comprehensive test analysis report generated by the test result comprehensive analysis model is not merely a listing of facts, but also reveals potential root causes and systemic risks through association analysis, providing a clear direction for subsequent remediation and optimization. Optionally, the comprehensive test analysis report may include a association analysis summary table, see Table 1.

[0033] Table 1: Example of a summary of multidimensional association analysis results In practical implementation, test optimization prompts for generative large models need to be carefully constructed to guide the model in generating structured output. The prompt template can include system instructions, contextual data, explicit task instructions, and output format examples. System instructions define the generative large model's role as a software test analyst. The contextual data provides a summary of the comprehensive test analysis report's conclusions in key-value pairs, code snippets of the current automated test script, and descriptions of relevant functionalities from the original test knowledge base. Task instructions explicitly require the generative large model to sequentially perform test coverage assessment, script problem diagnosis and correction suggestions, and new test task planning. The output format example can be a text structure with explicit chapter markers. In some embodiments, structured parsing of the generative large model's response text can utilize rule-based regular expression matching or pre-trained language models for information extraction. First, predefined chapter titles or keywords such as "##coverage assessment," "##script optimization suggestions," and "##new test task" are searched in the response text. Then, specific statements and list items are extracted within each chapter. It is understandable that iterative test optimization schemes are the key output connecting the current test round with the next test round. Based on historical test data and insights from comprehensive analysis, they generate actionable optimization suggestions through the extrapolation capabilities of generative large models, enabling test activities to continuously evolve and more effectively focus on high-risk and uncovered areas.

[0034] See Figure 4 This is a resource consumption analysis chart for a payment interface test, showing the trends of database connection count and interface response time during the test of the online payment gateway interface. The number of database connections rapidly increased from 50 at the start of the test, reaching a maximum of 100 in the 12th-14th minute and remaining saturated. It then gradually decreased, reflecting the resource exhaustion of the system under high load and the subsequent mitigation process. The red dotted line represents the maximum number of database connections (100), a critical threshold for resource bottlenecks. Interface response time is highly positively correlated with the number of database connections. When the number of connections reaches the maximum, the response time surges to 2.5 seconds, far exceeding the 2-second threshold. As the number of database connections decreases, the response time also decreases, verifying the causal relationship between the two.

[0035] In one embodiment of the present invention, the software automation testing method based on generative large models involves structured parsing of the response text of the generative large model to generate an iterative test optimization scheme. Assuming that after a test analysis of a user account management function, the response text returned by the generative large model contains the following paragraphs: "Evaluate existing test coverage: The current test covers user login, password modification, and basic information viewing functions. Covered boundary conditions include minimum password length verification and email format verification. Abnormal scenarios cover incorrect password login and session timeout re-login. Obvious omissions include scenarios such as session tokens should expire immediately after user logout and user attempts to modify other people's account information should be rejected. Optimize existing test scripts: In the automated test script for 'modifying user avatar,' there is a problem with the locator being invalidated. The script uses..." The CSS selector `#avatar-upload-btn` has been changed to `.profile-actions.upload-icon` in the current UI version. In the assertion for 'Checking Login Failure Lockout Policy', the assertion condition is inaccurate. The current script asserts that the account is immediately locked after three consecutive incorrect password attempts, but the actual business rule triggers locking only after the fifth incorrect attempt. It is recommended to change the number of incorrect attempts in the assertion condition from 3 to 5. It is recommended to add a test task: consider whether a later login session will invalidate the previous session when a user logs in simultaneously on different devices. System processing logic should be added to test when a user changes their account security email to an email address already associated with another account. A deduplication and error reporting mechanism should be designed to test batch import of user data when the imported file contains duplicate usernames. In the response text of the generative large model, locate the text paragraphs beginning with "Evaluate the existing test coverage as..." Parse the enumeration lists of covered functionalities, boundary conditions, and abnormal scenarios within these paragraphs, and identify any obvious omissions. From the example paragraphs above, the list of covered functionalities is "User login, password modification, basic information viewing"; the list of covered boundary conditions is "Minimum password length verification, email format verification"; the list of covered abnormal scenarios is "Incorrect password login, session timeout and re-login"; and the identified obvious omissions are "Session tokens should expire immediately after user logout, and attempts by users to modify other people's account information should be rejected." Organize the parsed information into the test coverage evaluation section of the iterative test optimization plan. In the response text of the generative large model, locate the text paragraphs beginning with "Optimize existing test scripts as..." Parse the descriptions of logical errors, invalid locators, and inaccurate assertions in the automated test scripts within these paragraphs, and extract the correction suggestions for each description. The parsed description regarding the inaccurate assertion condition is: "The current script asserts that the account is immediately locked after three consecutive incorrect password entries, but the actual business rule is that the lock is triggered only after the fifth incorrect entry." The suggested correction is: "Change the number of incorrect attempts in the assertion condition from 3 to 5." Pairing and organizing the above parsed description with the suggested correction results forms the modification suggestion section for the automated test script.

[0036] In the response text of the generative large model, locate the text paragraphs beginning with "It is suggested to add test tasks as follows," and analyze the new test scenarios, new data combinations, or new business processes proposed in these paragraphs. From the example paragraphs above, three suggestions are extracted: "Test whether a later login session will invalidate the previous session when a user logs in simultaneously on different devices," "Add system processing logic when a user changes their account security email to an email address already associated with another account," and "Design a deduplication and error reporting mechanism for batch importing user data when the imported file contains duplicate usernames." Analyze the core verification intent of each suggestion in the "Suggest adding test tasks" paragraph, and extract the core verification intent into a single test objective statement. For the first suggestion, "Test whether a later login session will invalidate the previous session when a user logs in simultaneously on different devices," its core verification intent is to verify the session mutual exclusion strategy. The test objective statement is extracted as "Verify that user accounts do not support multiple devices being online simultaneously, and that a later login session should invalidate the previous session." Regarding the second proposal, "Add system processing logic when a test user changes their account security email to an email address already associated with another account," the core verification intent is to verify the uniqueness constraint of email addresses. The test objective can be summarized as "Verify that the system prevents users from changing their security email to an email address already associated with another account." Regarding the third proposal, "Design a deduplication and error-reporting mechanism for batch importing user data when the imported file contains duplicate usernames," the core verification intent is to verify the handling of duplicate data during batch import. The test objective can be summarized as "Verify that the batch import user function can correctly handle duplicate usernames in the file and provide clear error messages."

[0037] Break down the implicit or explicit user operation sequences, system interactions, and data changes in each proposal, organizing them into a continuous series of steps from test preparation to result verification. For the test task of "verifying that user accounts do not support simultaneous online access from multiple devices," the steps are broken down as follows: Prepare a valid user account; successfully log in on device A using the account to obtain session token A; successfully log in on device B using the same account to obtain session token B; use session token A to access a personal information interface requiring authentication; check if the interface returns a message indicating session expiration or requiring re-login. For the test task of "verifying that the system prohibits users from changing their security email to an email address already associated with another account," the steps are broken down as follows: Prepare two existing user accounts, User1 and User2, where User2's security email is mailB@example.com; log in to the system using User1's credentials; in the account settings, attempt to change User1's security email to mailB@example.com; submit a modification request; check if the system rejects the modification and returns a message indicating that the email address is already in use. Assign a step number to each step and ensure logical coherence between steps to form a structured, concise step description. One method for assessing the logical coherence of step descriptions is to calculate the semantic dependency strength between steps, which can be expressed as: in: The score represents the logical coherence of the sequence of steps. Indicates the total number of steps. Indicates the first The semantic vector described by each step Indicates the first The semantic vector described by each step It is to calculate the semantic vector of two consecutive steps. and The function representing the correlation between steps ranges from 0 to 1, with higher values ​​indicating tighter logical connections between steps. Assigning sequence numbers to steps and ensuring logical coherence helps generate clear, executable task descriptions, avoiding contradictory step orders or missing preconditions. Pairing the test objective statement with a brief step description creates a complete test task entry. For example, pairing the test objective statement "verify that user accounts do not support multiple devices online simultaneously, and subsequent login sessions should invalidate previous sessions" with the brief step descriptions of the five steps above forms a complete test task entry to guide the next round of test script generation or manual test design. All test task entries together constitute the subsequent test task planning part of the iterative test optimization scheme.

[0038] See Figure 5This is a heatmap showing the correlation between test cases and defect types. The darker the color, the stronger the correlation. It's an important tool for defect risk distribution analysis in the comprehensive test analysis report phase. The email modification test has the strongest correlation with security defects, indicating that this test case is a key scenario for discovering security issues and has an extremely high risk level. The information viewing test has a relatively high correlation with functional defects, indicating significant stability issues with this functional module. The email modification test also has a medium-to-high correlation with compatibility and performance defects, requiring close attention. Most test cases have a low correlation with compatibility and interface defects, indicating that these two types of defects are less frequently exposed in the current test set.

[0039] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for automated software testing execution and result analysis based on generative large models, characterized in that: The method includes: Construct an original test knowledge base, extract information and fuse knowledge from the original test knowledge base, and generate a structured test knowledge graph; The test target description is extracted from the structured test knowledge graph, and the test target description, the preset test depth and the current software version information of the tested object are input into the generative large model. The output is a preliminary test script set including natural language test steps and expected result assertion descriptions. The natural language testing steps are converted into executable versions to generate machine-executable automated test scripts. At the same time, the expected result assertion descriptions are logically encoded to generate verifiable assertion verification rules. Deploy and execute the automated test script in the target test environment, collect the actual output data and system resource consumption records generated during the execution process, and form the original test execution record; The original test execution records are compared with the assertion verification rules to generate a preliminary result difference report. The preliminary result difference report is then analyzed using a defect pattern recognition model to generate a defect suspicion analysis report.

2. The method for automated software testing execution and result analysis based on generative large models according to claim 1, characterized in that, Information extraction and knowledge fusion are performed on the original test knowledge base to generate a structured test knowledge graph, including: Collect test task descriptions, product requirement documents, and historical test cases of the software under test to build an original test knowledge base; The structured test knowledge graph includes functional entities, test scenario entities, input / output constraint entities, and defect pattern entities, as well as their relationships. The test task description, product requirement document and historical test cases are parsed to identify text fragments related to functions, operations, data constraints and exception handling. Named entity recognition technology is applied to the identified text fragments to label the names of functional components, user behavior phrases, input data value ranges, and expected system response descriptions. Based on the pre-defined software testing domain ontology, the labeled functional component names, user behavior phrases, input data value ranges, and expected system response descriptions are mapped to instances of the functional entities, test scenario entities, input / output constraint entities, and defect mode entities. Based on the software business process logic and the step logic of test cases, call, containment, pre-requirement, post-requirement and equivalence class relationships are established among the identified entity instances to complete knowledge fusion and construct the structured test knowledge graph.

3. The method for automated software testing execution and result analysis based on generative large models according to claim 2, characterized in that, The natural language testing steps are converted into executable versions to generate machine-executable automated test scripts, including: Select the appropriate automated testing framework for the application type to which the object under test belongs; The core actions, operation objects, and input data in the natural language testing steps are analyzed. The core actions are mapped to the standard operation functions of the automated testing framework, the operation objects are located as element identifiers supported by the automated testing framework, and the input data is encapsulated into data parameters that meet the requirements of the function interface. Based on the order of the natural language testing steps described, the mapped standard operation functions, element identifiers, and data parameters are arranged and combined, and necessary synchronization wait and exception handling control logic is inserted to compile and generate the automated test script that can be executed by the test engine.

4. The method for automated software testing execution and result analysis based on generative large models according to claim 3, characterized in that, Simultaneously, the expected result assertion description is logically encoded to generate verifiable assertion verification rules, including: From the description of the expected result assertion, the expected output result to be verified, the verification condition, and the verification logic are separated. The expected output result is transformed into a specific expected value, which can be text, numerical value, Boolean value or complex data structure; The verification conditions are translated into comparison operators, including equal to, contain, greater than, and between; By combining the verification logic with expected values ​​and comparison operators, and using the syntax structure of the assertion programming framework, the assertion verification rules that can automatically verify the actual output data at runtime are constructed.

5. The method for automated software testing execution and result analysis based on generative large models according to claim 4, characterized in that, The original test execution records are compared with the assertion verification rules to generate a preliminary result difference report, including: Under the control of the test execution agent, the actual output data that needs to be verified at the assertion time is extracted from the original test execution record; According to the execution order of the assertion verification rules, the actual output data is matched and calculated with the expected value and comparison operator defined in the assertion verification rules in turn; When the comparison result between the actual output data and the expected value does not meet the conditions defined by the comparison operator, the assertion verification failure is recorded, and the time of failure, the test steps involved, the actual output data, the expected value, and the comparison operator used are recorded as a difference record. All discrepancies generated during the execution of a complete test script are summarized and accompanied by execution environment information to form a structured preliminary result discrepancy report.

6. The method for automated software testing execution and result analysis based on generative large models according to claim 5, characterized in that, The analysis of the preliminary result difference report using a defect pattern recognition model to generate a defect suspicion analysis report includes: Based on a historical defect database, the defect pattern recognition model is trained, and the defect pattern recognition model can identify the difference features that match known defect patterns; Extract the difference pattern features from the preliminary result difference report. The difference pattern features include the position of the difference in the test process, the data types involved, the keywords of the error message, and the difference chain. The difference pattern features are input into the trained defect pattern recognition model, and the defect pattern recognition model outputs the matching probability of each difference record with each known defect pattern. For each discrepancy record, the defect pattern with the highest matching probability is selected as the candidate defect cause. All discrepancy records are sorted according to the matching probability, and the sorting results are associated with the candidate defect causes to generate the defect suspicion analysis report.

7. The method for automated software testing execution and result analysis based on generative large models according to claim 6, characterized in that, Also includes: The preliminary result difference report, the defect suspicion analysis report, and the system resource consumption record in the original test execution record are input together into the test result comprehensive analysis model to generate a comprehensive test analysis report. The comprehensive test analysis report, the original test knowledge base, and all data collected in the current test round are input into the generative large model. The generative large model then outputs an iterative test optimization scheme that includes test coverage assessment, test script optimization suggestions, and subsequent test task planning. The generation of the comprehensive test analysis report specifically includes: The test result comprehensive analysis model receives the preliminary result difference report, defect suspicion analysis report and system resource consumption record as input; The comprehensive analysis model of the test results contains a multi-dimensional correlation analysis network, which simultaneously analyzes differences at the functional level, root causes of defects, and anomalies at the resource level, and seeks correlation patterns among these differences, root causes of defects, and anomalies at the resource level. When a strong temporal correlation is detected between a specific functional difference, a specific candidate defect cause, and a specific abnormal pattern of system resource consumption, the multi-dimensional correlation analysis network marks this situation as a high-confidence defect correlation event. Summarize all the difference analysis results and defect suspicion analysis results, and integrate the high-confidence defect association events to generate a comprehensive test analysis report that includes test conclusions, defect risk distribution, and performance bottleneck indications.

8. The method for automated software testing execution and result analysis based on generative large models according to claim 7, characterized in that, Also includes: The comprehensive test analysis report, the original test knowledge base, and all data collected in the current test round are input into the generative large model. The generative large model outputs an iterative test optimization scheme that includes test coverage assessment, test script optimization suggestions, and subsequent test task planning, specifically including: The comprehensive test analysis report, the original test knowledge base, the automated test scripts executed in the current test round, the original test execution records, the preliminary result difference report, and the defect suspicion analysis report are integrated and organized into structured multi-round test context information; The multi-round test context information and the background information of software iterative development are encapsulated together to form test optimization prompt information for generative large models; The test optimization prompts are sent to the generative big model, which is then instructed to focus on evaluating existing coverage, identifying missing scenarios, correcting existing script defects, and planning subsequent tests. The system receives the response text from the generative large model regarding the test optimization prompts, performs structured parsing on the response text, extracts information on test coverage assessment, suggestions for modifying existing automated test scripts, and subsequent test task planning for uncovered functions or scenarios, and organizes the extracted content into the iterative test optimization scheme.

9. The method for automated software testing execution and result analysis based on generative large models according to claim 8, characterized in that, The structured parsing of the response text extracts information on test coverage assessment, suggestions for modifying existing automated test scripts, and plans for subsequent test tasks for uncovered functions or scenarios, including: In the response text, locate the text paragraph that begins with an assessment of the existing test coverage, parse the enumeration list of covered functional points, boundary conditions, and abnormal scenarios in the text paragraph, identify the obvious omissions, and organize the above information into the test coverage assessment section. In the response text, locate the text paragraph that begins with "Optimize existing test scripts", parse the descriptions in the text paragraphs that indicate logical errors, invalid locators, and inaccurate assertion conditions in the automated test scripts, and extract the correction suggestions for each description and organize them into the modification suggestion section for the automated test scripts. In the response text, locate the text paragraphs that begin with a suggestion to add new test tasks, parse the new test scenarios, new data combinations or new business processes proposed in the text paragraphs, and transform each suggestion into a test task item containing test objectives and a brief description of steps, and organize it into the subsequent test task planning section.

10. The method for automated software testing execution and result analysis based on generative large models according to claim 9, characterized in that, The process of transforming each proposal into a test task entry that includes a test objective and a brief description of steps includes: Analyze the core verification intent of each proposal in the suggested new test task paragraph, and extract the core verification intent into a test objective statement; Deconstruct the implicit or explicit user action sequences, system interactions, and data changes in each proposal, and organize them into a continuous series of steps from test preparation to result verification; Assign a step number to each step and ensure logical coherence between steps to form a structured and concise step description; Pair the test objective statement with a brief step description to create a complete test task entry, which will guide the generation of the next round of test scripts or manual test design.

Images