An automated interface testing method and system based on a constraint-aware large model
By adopting an interface automation testing method based on a constraint-aware large model, the problems of high cost and low participation in interface testing of complex backend systems are solved. It achieves efficient and accurate interface test case generation and visual reports, and supports rapid feedback in continuous integration/continuous delivery processes.
Patent Information
- Application Number
- CN202611428582.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies for interface testing of complex backend systems suffer from problems such as high costs of test case construction and maintenance, technical barriers limiting test participation, 'illusion' and technical gaps in model application, and poor controllability and traceability of the testing process, making it difficult to support the rapid feedback requirements in continuous integration/continuous delivery processes.
An interface automation testing method based on a constraint-aware large model is adopted. By receiving natural language testing requirements, semantic analysis is performed using a domain-fine-tuned pre-trained language model to generate structured test cases that conform to interface constraints. Multi-interface dependencies are handled through slot filling strategies and cached reference mechanisms to achieve full-process automation and visualization report generation.
It improves the accuracy and efficiency of test case generation, reduces test case maintenance costs, increases the participation of non-technical personnel, realizes end-to-end scenario-based testing of complex systems and controllable and traceable testing processes, and supports rapid feedback in continuous integration/continuous delivery processes.
Smart Images

Figure CN123114976A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of software testing technology, and in particular relates to an interface automated testing method and system based on a constraint-aware large model. Background Technology
[0002] As enterprises deepen their digital transformation, the number of interfaces in complex backend systems is growing exponentially. Interface testing for these systems not only needs to cover routine input / output validation, but also needs to simulate cross-interface data association and multi-source data collaboration scenarios, posing a severe challenge to automated testing technologies.
[0003] Existing software testing technologies typically suffer from the following shortcomings:
[0004] (1) High cost of test case construction and maintenance: In actual engineering, a single interface usually requires dozens or even hundreds of test cases to cover various scenarios such as positive examples, negative examples, and boundary values. When the interface changes (such as adding parameters or adjusting paths), testers need to locate and modify the relevant test cases one by one, which is extremely inefficient and easy to miss. The maintenance cost increases linearly or even superlinearly with the scale of test cases.
[0005] (2) Technical barriers limit test participation: Traditional automated testing requires testers to have multiple technical skills such as Python programming, YAML syntax, and pytest framework. Business testers or product personnel without technical backgrounds cannot directly participate in the writing and maintenance of automated tests, resulting in low team manpower utilization and the promotion of automated testing highly dependent on a few technical personnel.
[0006] (3) The application of models suffers from "illusion" and technological gaps: In recent years, large language models have shown great potential in the field of code generation. However, when they are directly applied to test case generation, there are significant technological gaps, including:
[0007] The "illusion" problem: General-purpose models lack a firm understanding of interface parameter constraints (such as data types, value ranges, required fields, and enumeration values), easily generating "illusion" data that doesn't conform to the interface definition. For example, generating a negative number for a parameter that requires a positive integer, or generating a value outside the enumeration range for an enumeration type parameter;
[0008] Limited Scenario: Existing solutions mostly focus on generating test cases for a single interface, failing to handle data dependencies between multiple interfaces and unable to generate end-to-end scenario-based test cases;
[0009] Process fragmentation: Existing solutions typically only cover the isolated "test case generation" step, failing to form a closed loop with subsequent "test case maintenance," "test execution," and "results analysis," resulting in limited overall automation.
[0010] (4) Poor controllability and traceability of the testing process: Existing testing tools lack unified quantitative analysis capabilities, the test coverage cannot be intuitively evaluated, and after test case execution failures, it is necessary to manually check the lengthy logs one by one. Problem localization and root cause analysis are inefficient and cannot support the rapid feedback requirements in the continuous integration / continuous delivery (CI / CD) process. Summary of the Invention
[0011] The purpose of this invention is to provide an interface automation testing method and system based on a constraint-aware large model, aiming to solve the technical problems existing in the prior art mentioned in the background.
[0012] The embodiments of the present invention are implemented as follows:
[0013] An automated interface testing method based on a constraint-aware large model, the method specifically includes the following steps:
[0014] S1. Receive the natural language testing requirements input by the testers, and use the pre-trained language model that has been fine-tuned for the domain to perform semantic analysis, and decompose the natural language testing requirements into one or more testing intentions.
[0015] S2. For one or more interfaces involved in the test intent, parse the corresponding standard interface document, extract the constraint information of multiple parameters, and construct a parameter constraint matrix.
[0016] S3. The parameter constraint matrix is fused and used as input to the pre-trained language model that has been fine-tuned in the domain. The structured test cases that conform to the interface constraints are generated through the slot filling strategy.
[0017] S4. Dynamically convert the structured test cases into parameterized test scripts, execute them in the order of dependencies, collect execution logs, calculate the test pass rate metric, and generate a visual test report.
[0018] As a further limitation of the technical solution of this embodiment of the invention, the test intent includes an interface identifier, a test scenario type, and a description of expected behavior.
[0019] As a further limitation of the technical solution of the embodiment of the present invention, the parameter constraint matrix is arranged with parameters as rows and constraint dimensions as columns; the constraint dimensions include at least data type, whether it is required, and value range.
[0020] As a further limitation of the technical solution of this embodiment of the invention, S2 further includes: calculating the importance weight of each parameter in the parameter constraint matrix using the entropy weight method, specifically:
[0021] The parameter constraint matrix is subjected to data standardization processing, and the information entropy of multiple constraint dimensions is calculated;
[0022] Based on the multiple information entropies, determine the dimension weights of multiple constraint dimensions;
[0023] Calculate the combined weight value of multiple parameters based on the weights of the multiple dimensions.
[0024] As a further limitation of the technical solution of this embodiment of the invention, the calculation formula for data standardization is as follows:
[0025] ;
[0026] in, The standardized weight value represents the weight of the first... The parameter in the first... The proportion of the values under each constraint dimension to the total sum of that constraint dimension; For the first The parameter in the first... Original measurements or scores under each constraint dimension; Total One parameter;
[0027] The formula for calculating the information entropy is:
[0028] ;
[0029] in, For the first Information entropy of each constraint dimension; This is the adjustment coefficient;
[0030] The formula for calculating the dimensional weights is:
[0031] ;
[0032] in, For the first The dimension weights of each constraint dimension are: One constraint dimension;
[0033] The formula for calculating the comprehensive weight value is as follows:
[0034] ;
[0035] in, For the first The combined weight value of each parameter;
[0036] The higher the overall weight value, the more priority the corresponding parameters will have over constructing positive examples and boundary value test scenarios when generating structured test cases in S3.
[0037] As a further limitation of the technical solution of this embodiment of the invention, the slot filling strategy specifically includes:
[0038] When generating the values of each slot in the structured test cases, a beam search decoding algorithm is used. At each step of decoding, a preset number of k candidate sequences with the highest probability are retained. From all complete candidate sequences, the one with the highest overall probability score is selected as the output to ensure the logical coherence and syntactic correctness of the generated structured test cases. When generating test cases for multi-dependency scenarios, a cache setting field is automatically inserted into the upstream test case and a cache reference field is inserted into the downstream test case to achieve cross-interface data transfer.
[0039] As a further limitation of the technical solution of this embodiment of the invention, the training process of the pre-trained language model after domain fine-tuning is as follows:
[0040] Construct a training dataset that includes interface documentation descriptions and corresponding historical test cases;
[0041] Design a multi-task learning framework that includes two parallel tasks: parameter constraint extraction and test case sequence generation.
[0042] Based on the standard cross-entropy loss function, a constraint penalty term is introduced. When the parameter values generated by the model exceed the range defined by the parameter constraint matrix, the value of the constraint penalty term increases, thereby constraining the model's generation behavior during training and enabling the model to learn the parameter constraint rules.
[0043] As a further limitation of the technical solution of the embodiment of the present invention, S3 also includes:
[0044] Receive natural language modification instructions input by testers;
[0045] The natural language modification instructions are parsed into structured operation instructions;
[0046] Traverse the set of use cases to be modified, and use the cosine similarity of the text embedding vectors to calculate the similarity between the modification instruction and each use case description in the set of use cases;
[0047] Use cases with a similarity greater than a preset threshold are identified as target use cases;
[0048] According to the operation instructions, perform corresponding field addition, deletion and / or modification operations on the target use case.
[0049] As a further limitation of the technical solution of the embodiment of the present invention, S4 also includes:
[0050] Collect error logs for all failed test cases;
[0051] The error log is converted into a high-dimensional vector using a text vectorization algorithm;
[0052] Cluster analysis is performed on the high-dimensional vectors to automatically group errors with the same or similar root causes into one category;
[0053] Generate a categorized summary report, showing the number of use cases and suspected root cause descriptions for each type of issue.
[0054] An interface automation testing system based on a constraint-aware large-scale model, the system comprising a natural language parsing module, a parameter constraint modeling module, a test case generation module, and a test execution and analysis module, wherein:
[0055] The natural language parsing module is used to receive natural language test requirements input by testers, and to perform semantic analysis using a pre-trained language model that has been fine-tuned for the domain, breaking down the natural language test requirements into one or more test intentions.
[0056] The parameter constraint modeling module is used to parse the corresponding standard interface documents, extract the constraint information of multiple parameters, and construct a parameter constraint matrix for one or more interfaces involved in test intents.
[0057] The test case generation module is used to fuse the parameter constraint matrix and use it as input to the pre-trained language model that has been fine-tuned in the domain, and generate structured test cases that conform to the interface constraints through the slot filling strategy.
[0058] The test execution and analysis module is used to dynamically convert the structured test cases into parameterized test scripts, execute them in the order of dependencies, collect execution logs, calculate test pass rate metrics, and generate visual test reports.
[0059] Compared with the prior art, the beneficial effects of the present invention are:
[0060] (1) The present invention generates high accuracy: By using the constraint injection mechanism and constraint penalty loss function, the interface parameter constraints are deeply integrated into the training and inference process of the model, which effectively solves the "illusion" problem when generating test cases for general large models;
[0061] (2) This invention can achieve full-process automation and low threshold: testers do not need to write any code, but only need to describe the test requirements in natural language. The system can automatically generate, execute and analyze test cases, enabling business testers without technical backgrounds to participate in automated testing, and significantly improving the team's manpower utilization rate.
[0062] (3) The present invention has low test case maintenance cost: the batch modification function based on natural language understanding and cosine similarity matching makes it possible for testers to automatically locate and batch update relevant test cases when the interface changes, simply by describing the modification intention in natural language. Compared with manual modification one by one, the maintenance efficiency is improved by about 80%.
[0063] (4) The present invention has strong scene adaptability: it can automatically identify the dependency relationship of the interface on multi-source heterogeneous data and automatically generate the corresponding data construction and calling logic. At the same time, it supports the scenario-based testing of multi-interface association through the caching reference mechanism, covering complex business scenarios that are difficult to handle by traditional tools.
[0064] (5) The testing process of this invention is controllable and traceable: Through quantitative indicators such as test case pass rate, interface pass rate, and average response time, and Allure visualization reports, the testing process is refined and controlled. Testers can quickly locate failed test cases and suspected root causes, and the efficiency of problem investigation is greatly improved. Attached Figure Description
[0065] Figure 1 A flowchart of the interface automated testing method based on a constraint-aware large model provided in an embodiment of the present invention is shown;
[0066] Figure 2 The diagram shows the structural block diagram of the interface automated testing system based on a constraint-aware large model provided in an embodiment of the present invention. Detailed Implementation
[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0068] It is understandable that existing software testing technologies typically have the following drawbacks: (1) High cost of test case construction and maintenance: In actual engineering, a single interface usually requires dozens or even hundreds of test cases to cover various scenarios such as positive examples, negative examples, and boundary values. When the interface changes (such as adding parameters or adjusting paths), testers need to locate and modify the relevant test cases one by one, which is extremely inefficient and easy to miss. The maintenance cost increases linearly or even superlinearly with the scale of test cases; (2) Technical barriers limit the participation of testers: Traditional automated testing requires testers to have multiple technical skills such as Python programming, YAML syntax, and pytest framework. Business testers or product managers without technical backgrounds often lack the necessary skills. Personnel cannot directly participate in the writing and maintenance of automated tests, resulting in low utilization of team manpower and the promotion of automated tests is highly dependent on a small number of technical personnel; (3) There is an "illusion" and technical gap in model application: In recent years, large language models have shown great potential in the field of code generation. However, when they are directly applied to test case generation, there is a significant technical gap; (4) Poor controllability and traceability of the test process: Existing test tools lack unified quantitative analysis capabilities, the test coverage cannot be intuitively evaluated, and after test case execution failure, it is necessary to manually check the lengthy logs one by one. Problem location and root cause analysis are inefficient and difficult to support the rapid feedback requirements in the continuous integration / continuous delivery (CI / CD) process.
[0069] To address the aforementioned issues, this invention discloses an automated interface testing method and system based on a constraint-aware large-scale model. The method receives natural language testing requirements input by testers, performs semantic analysis using a domain-fine-tuned pre-trained language model, and decomposes the natural language testing requirements into one or more testing intents. For the interfaces involved in one or more testing intents, the corresponding standard interface documents are parsed, and constraint information for multiple parameters is extracted to construct a parameter constraint matrix. The parameter constraint matrix is then fused and used as input to the domain-fine-tuned pre-trained language model. A slot-filling strategy is used to generate structured test cases that conform to the interface constraints. The structured test cases are dynamically converted into parameterized test scripts, executed in dependency order, execution logs are collected, test pass rate metrics are calculated, and a visual test report is generated. This method can transform the structured parameter constraints of the interface under test into a mathematical representation that the model can understand, and after fusion with the natural language testing requirements, input them into a task-fine-tuned pre-trained language model, thereby generating high-fidelity, directly executable test cases. This enables the automatic generation, batch maintenance, and execution analysis of test cases for complex system interfaces.
[0070] Specifically, Figure 1 The flowchart of the interface automation testing method based on a constraint-aware large model provided by an embodiment of the present invention is shown.
[0071] In a preferred embodiment of the present invention, an automated interface testing method based on a constraint-aware large model specifically includes the following steps:
[0072] Step S1: Receive the natural language testing requirements input by the testers, and use a pre-trained language model that has been fine-tuned for the domain to perform semantic analysis, breaking down the natural language testing requirements into one or more testing intentions.
[0073] In this embodiment of the invention, natural language testing requirements input by testers are received, and semantic analysis is performed using a pre-trained language model that has been fine-tuned for the domain. The natural language testing requirements are then decomposed into one or more independent testing intentions, each of which includes an interface identifier, a test scenario type, and a description of the expected behavior.
[0074] It is understood that, in this embodiment of the invention, by receiving the natural language testing requirements input by the tester (e.g., the natural language testing requirement is "Test file upload interface: 10MB PDF file successfully uploaded (positive example), assertion response code is 200"), semantic analysis is then performed using a domain-fine-tuned pre-trained language model to extract the following structured information:
[0075] Interface identifier: File upload interface;
[0076] Test scenario type: Positive example;
[0077] Key entities: file_type = PDF, file_size = 10MB;
[0078] Assert condition: response_code = 200;
[0079] For the pre-trained language model that has undergone domain fine-tuning, fine-tuning is performed using a multi-task learning framework, which includes two parallel tasks:
[0080] Task 1: Understanding Interface Parameters (Sequence Labeling Task):
[0081] Input: Natural language description text of the interface (e.g., "the file parameter is the binary file to be uploaded, with a size not exceeding 10MB");
[0082] Output: Labeled sequences, identify and extract structured entities such as [parameter name], [data type], [required / unrestricted], and [value range];
[0083] Objective: To enable the model to learn to extract structured parameter constraint information from unstructured text.
[0084] Task 2: YAML Sequence Generation (Sequence Generation Task)
[0085] Input: The parameter constraint structure extracted from Task 1 (e.g., {name: file, type: binary, max_size: 10MB, required: true}).
[0086] Output: Test case code snippets conforming to YAML syntax;
[0087] Objective: To enable the model to learn to translate structured constraints into standardized YAML test cases.
[0088] The fine-tuning dataset originates from a standard test case library manually written by test experts at the beginning of the project, or from existing historical test cases exported from the existing automated testing framework. The fine-tuning dataset includes a parameter understanding dataset, which extracts the parameter descriptions of the interfaces from the OpenAPI documentation, manually annotates the corresponding constraint entities, and constructs "description-annotation" pairs; and a test case generation dataset, which collects historical manually written YAML test cases, pairs them with the corresponding interface parameter constraints, and constructs "constraint structure-YAML fragment" pairs.
[0089] To ensure the compliance of the model's generated results, a constraint penalty term is introduced on top of the standard cross-entropy loss function, resulting in a total loss function. for:
[0090] ;
[0091] in, The cross-entropy loss for Task 1 measures the accuracy of the model in labeling parameter-constrained entities. For Task 2, the cross-entropy loss measures the difference between the YAML sequence generated by the model and the real use case. λ1 and λ2 are the constraint penalty terms; they are the balancing coefficients used to adjust the relative weights of the three losses.
[0092] By introducing constraint penalty terms, the model can be "penalized" during training to generate non-compliant parameter values, thereby learning the constraint rules internally and effectively reducing the "illusion" problem in the inference stage.
[0093] After fine-tuning, verify whether the model meets the usability standard using the following two metrics. Both metrics must be met simultaneously; otherwise, additional training data needs to be added and fine-tuned again. Specifically:
[0094] The first item, parameter comprehension accuracy, is calculated using the following formula:
[0095] ;
[0096] in, For parameter understanding accuracy; Constrain the number of entities to ensure the correct extraction of parameters from the model; The total number of entities constrained by parameters in the test set;
[0097] The second item, YAML generation compliance rate, is calculated using the following formula:
[0098] ;
[0099] in, Generate compliance rates for YAML; The number of test cases generated for the model that can be correctly parsed by the YAML parser and pass syntax validation; This represents the total number of test cases generated.
[0100] Step S2: For one or more interfaces involved in the test intent, parse the corresponding standard interface document, extract the constraint information of multiple parameters, and construct a parameter constraint matrix.
[0101] In this embodiment of the invention, for one or more interfaces involved in testing intent, the standard interface document is automatically parsed to extract the constraint information of each parameter, construct a parameter constraint matrix, and calculate the importance weight of each parameter in the parameter constraint matrix using the entropy weight method. Specifically, the parameter constraint matrix is subjected to data standardization processing, the information entropy of multiple constraint dimensions is calculated, and then the dimension weights of multiple constraint dimensions are determined based on the multiple information entropies. Finally, the comprehensive weight value of multiple parameters is calculated based on the multiple dimension weights. The parameter constraint matrix is arranged with parameters as rows and constraint dimensions as columns, and each constraint dimension includes at least the data type, whether it is required, and the value range. Specifically, the calculation formula for data standardization is as follows:
[0102] ;
[0103] in, The standardized weight value represents the weight of the first... The parameter in the first... The proportion of the values under each constraint dimension to the total sum of that constraint dimension; For the first The parameter in the first... Original measurements or scores under each constraint dimension; Total One parameter;
[0104] The formula for calculating the information entropy is:
[0105] ;
[0106] in, For the first Information entropy of each constraint dimension; This is the adjustment coefficient;
[0107] The formula for calculating dimensional weights is:
[0108] ;
[0109] in, For the first The dimension weights of each constraint dimension are: One constraint dimension;
[0110] The formula for calculating the overall weight value is:
[0111] ;
[0112] in, For the first The combined weight value of each parameter;
[0113] The higher the overall weight value, the more priority the corresponding parameters will have over constructing positive examples and boundary value test scenarios when generating structured test cases in S3.
[0114] It is understood that, in this embodiment of the invention, the unstructured interface definition document is transformed into structured mathematical knowledge that can be directly consumed by the model. Specifically:
[0115] The API documentation parsing process includes:
[0116] (1) Document loading: Read the OpenAPI 3.0 document and extract the list of interfaces under the paths field. Each interface contains the request path, request method, parameter list, etc.
[0117] (2) Parameter extraction: For each parameter, extract the following attributes: parameter name, parameter position, parameter type, whether it is required, enumeration value (if any), string length constraint (if any);
[0118] (3) Interface type differentiation: Determine whether the interface depends on multiple source data (such as calling the PG library, MinIO document, Jena triplet library). If the parameter contains a file field (with the value PG / MinIO / Jena / PG_Spatial), it is a "related dataset interface"; otherwise, it is a "non-related dataset interface".
[0119] In constructing the parameter constraint matrix, the extracted parameter constraints are transformed into a mathematical matrix to facilitate subsequent model invocation and test case generation. This process defines the parameter constraint matrix. ,in The number of parameters contained in the interface; Constraint dimensions (e.g., when k=5, set the corresponding constraints as: required, data type, minimum value, maximum value, data source dependency); matrix elements Representing the The parameter of the first The constraint values and their selection rules are shown in the table below:
[0120] Constraint dimension j Constraints Value selection rules 1 Is this field required? Yes = 1 No = 0 2 Data types string=1int=2float=3file=4boolean=5enum=6 3 Minimum value Enter a specific value for numeric types, and enter 0 for non-numeric types. 4 Maximum value Enter a specific value for numeric types, and enter 0 for non-numeric types. 5 Data source dependence Related datasets = 1 (PG / MinIO, etc.) Unrelated datasets = 0
[0121] Assuming the file upload interface has three parameters: file, file_name, and file_size, where the file_name parameter has a maximum string length of 256 characters and the file_size parameter has a maximum file size of 50MB, the parameter constraint matrix for the knowledge platform's "File Upload Interface" (associated with MinIO) is as follows:
[0122] ;
[0123] The first line corresponds to the file parameter (required, file type, no minimum / maximum constraints, depends on external data source), the second line corresponds to the file_name parameter (required, string type, no minimum length, maximum length 256, does not depend on external data source), and the third line corresponds to the file_size parameter (required, integer type, minimum value 0, maximum value 52428800, i.e. 50MB, does not depend on external data source).
[0124] To prioritize coverage of core business parameters during test case generation, the entropy weight method can be used to objectively calculate the importance weight of each parameter. The core idea of the entropy weight method is that the greater the difference between different parameters of a constraint dimension (i.e., the smaller the information entropy), the more discriminative information that dimension provides, and the higher its corresponding weight should be. Specifically, the calculation steps are as follows:
[0125] (1) Data standardization: Each column of the constraint matrix M is standardized to eliminate the influence of dimensions. The formula for calculating data standardization is:
[0126] ;
[0127] in, The standardized weight value represents the weight of the first... The parameter in the first... The proportion of the values under each constraint dimension to the total sum of that constraint dimension; For the first The parameter in the first... Original measurements or scores under each constraint dimension; Total One parameter;
[0128] (2) Calculate the information entropy. The formula for calculating information entropy is:
[0129] ;
[0130] in, For the first Information entropy of each constraint dimension; This is the adjustment coefficient;
[0131] (3) Calculate the dimension weights. The formula for calculating the dimension weights is:
[0132] ;
[0133] in, For the first The dimension weights of each constraint dimension are: One constraint dimension;
[0134] (4) Calculate the overall weight of the parameters. The formula for calculating the overall weight value is:
[0135] ;
[0136] in, For the first The combined weight value of each parameter.
[0137] Step S3: The parameter constraint matrix is fused and used as input to the pre-trained language model that has been fine-tuned in the domain. The structured test cases that conform to the interface constraints are generated through the slot filling strategy.
[0138] In this embodiment of the invention, the natural language test intent is fused with the parameter constraint matrix of the corresponding interface and used as the input of the domain-fine-tuned pre-trained language model. The domain-fine-tuned pre-trained language model generates structured test cases that conform to the interface constraints based on a predefined test case skeleton template and through a slot filling strategy. When the natural language test requirement contains multiple test intents with data dependencies, the domain-fine-tuned pre-trained language model automatically identifies the dependencies between intents and establishes a data transmission link in the generated test cases through a caching reference mechanism.
[0139] The slot filling strategy specifically includes: when generating the values of each slot in the test case, a beam search decoding algorithm is used to retain a preset number of k candidate sequences with the highest probability at each step of the decoding, and finally selects the one with the highest overall probability score from all complete candidate sequences as the output to ensure the logical coherence and syntactic correctness of the generated test cases;
[0140] The process then includes a batch modification step for test cases: receiving natural language modification instructions input by testers and parsing the modification instructions into structured operation instructions; traversing the set of test cases to be modified, calculating the similarity between the modification instructions and each test case description using the cosine similarity of the text embedding vectors, and identifying test cases with a similarity greater than a preset threshold as target test cases; and performing corresponding field addition, deletion, and modification operations on the target test cases according to the operation instructions.
[0141] Understandably, in this embodiment of the invention, regarding constraint fusion and input construction, before fusion, the parameter constraint matrix M is first serialized, transforming it into a structured natural language description text or JSON-formatted constraint information. The extracted entities are then matched and verified against the parameter constraint matrix M. For example, it is verified whether file_type = PDF is within the enumeration range defined by the interface, and whether file_size = 10MB is within the range [0, 50MB]. After successful verification, the natural language intent is serialized into structured text C, concatenated with the original intent text T, and the final input of the model is constructed.
[0142] ;
[0143] For template-based slot filling and bundle search decoding, the model does not generate YAML text from scratch, but fills slots based on a predefined YAML skeleton template. In one embodiment of this invention, the skeleton template for a "positive example of a POST request" is as follows:
[0144] case_name}:
[0145] host: ${host()}
[0146] url: {url_slot}
[0147] method: POST
[0148] detail: {detail_slot}
[0149] headers:
[0150] Content-Type: application / json;charset=UTF-8
[0151] token: $cache{login_token}
[0152] requestType: {request_type_slot}
[0153] data: {data_slot}
[0154] assert:
[0155] code:
[0156] jsonpath: $.code
[0157] type: ==
[0158] value: {assert_code_slot}
[0159] When filling each slot, the model employs a beam search decoding algorithm. Unlike greedy search (which selects only the word with the highest probability at each step), beam search retains k candidate sequences with the highest probabilities at each decoding step, and finally selects the one with the highest overall probability score from all complete candidate sequences as the output. This effectively avoids the local optima that greedy search may get stuck in, improving the logical coherence and syntactic correctness of the generated use cases;
[0160] For the core fields of the generated YAML test cases, the YAML test cases must conform to the pytest framework parsing specifications, as shown in the table below:
[0161] English field names Chinese explanation Is this field required? Example value name Use case unique identifier yes file_upload_positive_10MB_PDF host Target server address yes ${host()} url API request path yes / api / v2 / fileManage / uploadFile method HTTP request methods yes POST detail Use case description, explaining the purpose of the test. yes Knowledge Management - Data Management - Upload PDF Files headers Request header information no Content-Type: application / json requestType Request body type no json / file data Request parameters (key-value pairs) yes file: ${minio.get_file(...)} assert Assertion rules (supports multiple dimensions) yes code: 200, message: Execution successful current_request_set_cache Caching settings are used for data transfer between interfaces. no jsonpath: $.detail, name: file_id
[0162] It also supports assertion validation on multiple dimensions of the response results, as shown in the following example format:
[0163] assert:
[0164] code: # Assertion Dimension 1: Response Status Code
[0165] jsonpath: $.code # Use JSONPath to extract response fields
[0166] type: == # Comparison operators (==, !=, contains, regex, etc.)
[0167] value: 200 # Expected value
[0168] message: # Assertion Dimension 2: Response Message
[0169] jsonpath: $.message
[0170] type: ==
[0171] value: Execution successful
[0172] data_id: # Assertion Dimension 3: Response Data Field
[0173] jsonpath: $.detail.id
[0174] type: !=
[0175] value: null
[0176] For interfaces that rely on external data sources (such as those that need to retrieve test files from MinIO), data retrieval logic can be automatically embedded in the data field:
[0177] ata:
[0178] file:
[0179] file: ${minio.get_file(file_type="PDF", file_size=10)}
[0180] file_name: "test_10mb.pdf"
[0181] For multi-interface scenarios with data dependencies, a data transmission chain is established through the `current_request_set_cache` field. The upstream interface stores key return values in a cache, and the downstream interface references them via `$cache{variable_name}`.
[0182] # Upstream use case: Creating an ontology
[0183] create_ontology: ...
[0185] current_request_set_cache:
[0186] - type: response
[0187] jsonpath: $.content[0].url
[0188] name: ontology_id_material
[0189] # Downstream use case: Creating a relationship (referencing the upstream cache)
[0190] create_relation: ...
[0192] data:
[0193] sourceUrl: $cache{ontology_id_material}
[0194] Step S4: Dynamically convert the structured test cases into parameterized test scripts, execute them in the order of dependencies, collect execution logs, calculate the test pass rate index, and generate a visual test report.
[0195] In this embodiment of the invention, the generated structured test cases are dynamically converted into parameterized test scripts executable by the automated testing framework, executed in dependency order, execution logs are collected, test pass rate indicators are calculated, and a visual test report is generated. Furthermore, root cause analysis based on log clustering is performed: by collecting error logs from all failed test cases, the logs are converted into high-dimensional vectors using a text vectorization algorithm; cluster analysis is performed on all failed log vectors, automatically grouping errors with the same or similar root causes into one category; a classification summary report is generated, displaying the number of test cases and suspected root cause descriptions for each type of problem.
[0196] It is understood that, in this embodiment of the invention, the following example of generating a pytest script is provided:
[0197] import allure
[0198] import pytest
[0199] from utils.read_files_tools.get_yaml_data_analysis import GetTestCase
[0200] from utils.assertion.assert_control import Assert
[0201] from utils.requests_tool.request_control import RequestControl
[0202] # Load test data from a YAML file
[0203] case_id = ['file_upload_positive_10MB_PDF', 'file_upload_negative_exceed_size', ...]
[0204] TestData = GetTestCase.case_data(case_id)
[0205] re_data = regular(str(TestData))
[0206] @allure.epic("Knowledge Platform System")
[0207] @allure.feature("Data Management Module")
[0208] class TestFileUpload:
[0209] @allure.story("File upload functionality")
[0210] @allure.title("{name}")
[0211] @pytest.mark.parametrize(
[0212] 'in_data',
[0213] eval(re_data),
[0214] ids=[i['detail'] for i in TestData] )
[0216] def test_file_upload(self, in_data):
[0217] """
[0218] Automated testing of file upload interface
[0219] :param in_data: Single test case data parsed from YAML
[0220] """
[0221] # Send HTTP request
[0222] res = RequestControl(in_data).http_request()
[0223] # Perform multi-dimensional assertions
[0224] Assert(
[0225] assert_data=in_data['assert_data'],
[0226] response_data=res.response_data
[0227] ).assert_type_handle()
[0228] During test execution, the pytest framework automatically generates standardized logs corresponding to the test scenario, including the test case collection phase, the test case execution phase, and the execution result summary. The test logs can provide real-time feedback on the execution order and status of each business step, and also provide structured raw data for the generation of test reports.
[0229] Next, by collecting the raw logs generated by pytest execution, quantitative analysis is performed to generate a visualized Allure test report. The core metrics calculation includes:
[0230] (1) Test case pass rate: measures the success rate of test cases, and is calculated using the following formula:
[0231] ;
[0232] in, Test case pass rate; The number of test cases where all assertions pass; This represents the total number of test cases executed.
[0233] (2) Interface pass rate: measures the overall stability of the interface, and is calculated using the following formula:
[0234] ;
[0235] in, For interface pass rate;
[0236] (3) Average response time: measures the performance of the interface, and is calculated using the following formula:
[0237] ;
[0238] in, This is the average response time; For the first Response time for each use case; This represents the total number of test cases executed.
[0239] (4) High-frequency error test cases: Statistically count the top cases with the most errors The formula for calculating the number of use cases is:
[0240] ;
[0241] in, for; To test the number of executions; This is an indicator function (1 if the test case fails, 0 if it succeeds); Indicates the event determination condition, in the first... During this run, the test cases Whether the execution failed; if it failed, return 1; if it succeeded, return 0.
[0242] For Allure report generation, the generated Allure report presents test results in a multi-dimensional manner, detailing the execution logs, request and response data, assertion results, and failed test case markers for each test case. Furthermore, the report supports grouping test case results by interface and sorting test cases based on "response time," helping to quickly pinpoint performance bottleneck interfaces. It also displays the pass rate trend across multiple tests; trend analysis can determine the stability of the interface, providing strong support for continuous integration and continuous delivery.
[0243] Furthermore, Figure 2 The diagram shows the structural block diagram of the interface automated testing system based on a constraint-aware large model provided in an embodiment of the present invention.
[0244] Specifically, in another preferred embodiment provided by the present invention, an interface automation testing system based on a constraint-aware large model includes:
[0245] The natural language parsing module 100 is used to receive natural language testing requirements input by testers, and to perform semantic analysis using a pre-trained language model that has been fine-tuned for the domain, breaking down the natural language testing requirements into one or more testing intentions.
[0246] The parameter constraint modeling module 200 is used to parse the corresponding standard interface documents, extract the constraint information of multiple parameters, and construct a parameter constraint matrix for one or more interfaces involved in test intent.
[0247] The test case generation module 300 is used to fuse the parameter constraint matrix and use it as input to the pre-trained language model that has been fine-tuned in the domain, and generate structured test cases that conform to the interface constraints through a slot filling strategy.
[0248] The test execution and analysis module 400 is used to dynamically convert the structured test cases into parameterized test scripts, execute them in the order of dependencies, collect execution logs, calculate test pass rate indicators, and generate visual test reports.
[0249] In a specific embodiment of this invention, taking the testing of "knowledge platform ontology model management" as an example, the knowledge platform is a complex backend system that integrates multi-source heterogeneous data such as structured databases, unstructured documents, and knowledge graph engines. Its interface testing needs to handle complex data associations and multi-source data call scenarios, which is a typical applicable scenario of this invention. Specifically:
[0250] (1) Scene preparation:
[0251] Test objective: To automate the testing of the "ontology model management" function of the knowledge platform, covering ontology creation, relationship establishment, abnormal scenario verification, and relationship editing operations.
[0252] Input requirement: Testers need to input a complex natural language command:
[0253] Please generate test cases for the ontology management function: First, create an ontology named "Raw Materials", then create a "Chemical Reaction" relationship based on it that points to "Products"; next, try to repeatedly create the "Raw Materials" ontology to verify the system's uniqueness check; finally, rename the "Chemical Reaction" relationship to "Synthetic Reaction".
[0254] Interface documentation: The system backend has integrated the OpenAPI 3.0 documentation for this module, which defines detailed parameter constraints for interfaces such as addOntologies (create ontologies), addOntologiesSide (create relationships), and editOntologiesRel (edit relationships);
[0255] (2) Execution steps:
[0256] Step 1: Natural Language Parsing and Intent Decomposition
[0257] Upon receiving the aforementioned long sentence request, semantic analysis was performed using a fine-tuned BERT model, breaking it down into four independent test intents:
[0258] meaning Figure 1 (Example): Create an ontology (name=raw material). Corresponding interface: addOntologies;
[0259] meaning Figure 2 (Positive Example / Relationship): Create a relationship (Source = Raw Material, Target = Product, Type = Chemical Reaction). Corresponding Interface: addOntologiesSide;
[0260] Intent 3 (Counterexample): Create an ontology (name=raw material, expected result=failure / already exists). Corresponding interface: addOntologies;
[0261] Intent 4 (Modification): Edit relationships (old name = chemical reaction, new name = synthetic reaction). Corresponding interface: editOntologiesRel;
[0262] Step 2: Parameter Constraint Modeling and Context Construction
[0263] For Italy Figure 1: Parse the OpenAPI documentation of the addOntologies interface and construct a parameter constraint matrix; extract the parameter constraints: url (String, required, maximum length 256), parent (String, optional); for the meaning Figure 2 : Parse the addOntologiesSide interface and extract the constraints: sourceUrl (String, required), targetUrl (String, required), and type (String, required).
[0264] Key point: Data transfer needs to be established in the use case through a caching reference mechanism;
[0265] Constructing the context: Serialize all constraint information into structured text (Context), concatenate it with the original natural language intent, and use it as input to the model;
[0266] Step 3: Constraint-aware use case generation:
[0267] Generate Use Case 1: Create an Ontology (corresponding to meaning) Figure 1 ):
[0268] The system recognizes that "raw materials" is an ontology name, automatically populates it into the url field, and caches the returned ontology ID as url_id_yuancailiao via current_request_set_cache for future use cases.
[0269] create_o2:
[0270] host: ${host()}
[0271] url: / api / giskg-sources / api / v2 / graphManager / addOntologies
[0272] method: POST
[0273] Detail: Ontology Management - Ontology Creation - Raw Materials
[0274] headers:
[0275] Content-Type: application / json;charset=UTF-8
[0276] token: $cache{login_token}
[0277] requestType: json
[0278] data:
[0279] URL: Raw Materials -- auto12315
[0280] assert:
[0281] code:
[0282] jsonpath: $.code
[0283] type: ==
[0284] value: 200
[0285] reason:
[0286] jsonpath: $.content[0].reason
[0287] type: ==
[0288] value: Insertion successful
[0289] current_request_set_cache:
[0290] - type: response
[0291] jsonpath: $.content[0].url
[0292] name: url_id_yuancailiao
[0293] Generate Use Case 2: Create Relationship (corresponding to meaning) Figure 2 ):
[0294] The model understands the relationship of "chemical reaction" by connecting "raw materials" and "products." It automatically references the cached url_id_yuancailiao from the previous step and generates a corresponding cached reference for the "products."
[0295] create_or2:
[0296] host: ${host()}
[0297] url: / api / giskg-sources / api / v2 / graphManager / addOntologiesSide
[0298] method: POST
[0299] Detail: Ontology Management - Relationship Creation - Chemical Reaction
[0300] headers:
[0301] Content-Type: application / json;charset=UTF-8
[0302] token: $cache{login_token}
[0303] requestType: json
[0304] data:
[0305] sourceUrl: $cache{url_id_yuancailiao}
[0306] targetUrl: $cache{url_id_chanwu}
[0307] type: Chemical reaction -- auto12315
[0308] assert:
[0309] code:
[0310] jsonpath: $.code
[0311] type: ==
[0312] value: 200
[0313] reason:
[0314] jsonpath: $.content[0].reason
[0315] type: ==
[0316] value: Ontology relationship added successfully
[0317] current_request_set_cache:
[0318] - type: response
[0319] jsonpath: $.content[0].type
[0320] name: type_id_huaxuefanying
[0321] Generate Test Case 3: Exceptional Scenario Testing (corresponding to Intent 3, counterexample):
[0322] For the intent of "repeated creation", the model generated the same data as in use case 1, but automatically adjusted the assertion logic, expecting the system to return an error message of "ontology already exists" instead of "insertion successful":
[0323] fcreate_o1:
[0324] host: ${host()}
[0325] url: / api / giskg-sources / api / v2 / graphManager / addOntologies
[0326] method: POST
[0327] Detail: Ontology Management - Duplicate Ontology Creation - Expected Failure
[0328] headers:
[0329] Content-Type: application / json;charset=UTF-8
[0330] token: $cache{login_token}
[0331] requestType: json
[0332] data:
[0333] URL: Raw Materials -- auto12315
[0334] assert:
[0335] code:
[0336] jsonpath: $.code
[0337] type: ==
[0338] value: 500
[0339] reason:
[0340] jsonpath: $.content[0].reason
[0341] type: contains
[0342] value: already exists
[0343] Generate use case 4: Editing operation (corresponding to intent 4):
[0344] Regarding the "rename" intent, the model recognized it as a PUT request and accurately populated oldType (referencing the previously cached relation type ID) and newType:
[0345] edit_or1:
[0346] host: ${host()}
[0347] url: / api / giskg-sources / api / v2 / graphManager / editOntologiesRel
[0348] method: PUT
[0349] Detail: Ontology Management - Edit Relationships - Rename
[0350] headers:
[0351] Content-Type: application / json;charset=UTF-8
[0352] token: $cache{login_token}
[0353] requestType: json
[0354] data:
[0355] oldType: $cache{type_id_huaxuefanying}
[0356] newType: Synthetic reaction --auto12315
[0357] assert:
[0358] code:
[0359] jsonpath: $.code
[0360] type: ==
[0361] value: 200
[0362] reason:
[0363] jsonpath: $.content[0].reason
[0364] type: ==
[0365] value: Ontology relationship edited successfully
[0366] Test execution and closed-loop verification include:
[0367] Script Conversion: Convert the above four YAML test cases into pytest scripts and inject test data using @pytest.mark.parameterize;
[0368] Sequential execution: The test engine executes test cases in the order of dependencies. After executing create_o2, the $.content[0].url in the response is automatically stored in the cache variable url_id_yuancailiao;
[0369] Related execution: When executing create_or2, $cache{url_id_yuancailiao} is automatically replaced with the actual ontology ID to ensure successful relationship creation;
[0370] Output results: All four test cases passed, and the Allure report clearly shows the execution status, response time, and assertion results for each test case.
[0371] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.
Claims
1. A method for interface automated testing based on constraint-aware large models, characterized in that, The method specifically includes the following steps: S1. Receive the natural language test requirements input by the testers, and use the pre-trained language model that has been fine-tuned for the domain to perform semantic analysis, and decompose the natural language test requirements into one or more test intentions. S2. For one or more interfaces involved in the test intent, parse the corresponding standard interface document, extract the constraint information of multiple parameters, and construct a parameter constraint matrix. S3. The parameter constraint matrix is fused and used as input to the pre-trained language model that has been fine-tuned in the domain. The structured test cases that conform to the interface constraints are generated through the slot filling strategy. S4. Dynamically convert the structured test cases into parameterized test scripts, execute them in the order of dependencies, collect execution logs, calculate the test pass rate metric, and generate a visual test report.
2. The constrained-aware large model based interface automated testing method according to claim 1, wherein, The test intent includes the interface identifier, test scenario type, and expected behavior description.
3. The constrained-aware large model based interface automated testing method of claim 1, wherein, The parameter constraint matrix is arranged with parameters as rows and constraint dimensions as columns; the constraint dimensions include at least the data type, whether it is required, and the range of values.
4. The constrained-aware large model based interface automated testing method according to claim 3, wherein, S2 further includes: calculating the importance weight of each parameter in the parameter constraint matrix using the entropy weight method, specifically: The parameter constraint matrix is subjected to data standardization processing, and the information entropy of multiple constraint dimensions is calculated; Based on the multiple information entropies, determine the dimension weights of multiple constraint dimensions; Calculate the combined weight value of multiple parameters based on the weights of the multiple dimensions.
5. The constrained-aware large model based interface automated testing method according to claim 4, wherein, The formula for calculating the data standardization is as follows: ; in, The standardized weight value represents the weight of the first... The parameter in the first... The proportion of the values under each constraint dimension to the total sum of that constraint dimension; For the first The parameter in the first... Original measurements or scores under each constraint dimension; Total One parameter; The formula for calculating the information entropy is: ; in, For the first Information entropy of each constraint dimension; This is the adjustment coefficient; The formula for calculating the dimensional weights is: ; in, For the first The dimension weights of each constraint dimension are: One constraint dimension; The formula for calculating the comprehensive weight value is as follows: ; in, For the first The combined weight value of each parameter; The higher the overall weight value, the more priority the corresponding parameters will have over constructing positive examples and boundary value test scenarios when generating structured test cases in S3.
6. The interface automated testing method based on a constraint-aware large model according to claim 1, characterized in that, The slot filling strategy specifically includes: When generating the values of each slot in the structured test cases, a beam search decoding algorithm is used. At each step of decoding, a preset number of k candidate sequences with the highest probability are retained. From all complete candidate sequences, the one with the highest overall probability score is selected as the output to ensure the logical coherence and syntactic correctness of the generated structured test cases. When generating test cases for multi-dependency scenarios, a cache setting field is automatically inserted into the upstream test case and a cache reference field is inserted into the downstream test case to achieve cross-interface data transfer.
7. The interface automated testing method based on a constraint-aware large model according to claim 1, characterized in that, The training process of the domain-fine-tuned pre-trained language model is as follows: Construct a training dataset that includes interface documentation descriptions and corresponding historical test cases; Design a multi-task learning framework that includes two parallel tasks: parameter constraint extraction and test case sequence generation. Based on the standard cross-entropy loss function, a constraint penalty term is introduced. When the parameter values generated by the model exceed the range defined by the parameter constraint matrix, the value of the constraint penalty term increases, thereby constraining the model's generation behavior during training and enabling the model to learn the parameter constraint rules.
8. The interface automated testing method based on a constraint-aware large model according to claim 1, characterized in that, S3 further includes: Receive natural language modification instructions input by testers; The natural language modification instructions are parsed into structured operation instructions; Traverse the set of use cases to be modified, and use the cosine similarity of the text embedding vectors to calculate the similarity between the modification instruction and each use case description in the set of use cases; Use cases with a similarity greater than a preset threshold are identified as target use cases; According to the operation instructions, perform corresponding field addition, deletion and / or modification operations on the target use case.
9. The interface automated testing method based on a constraint-aware large model according to claim 1, characterized in that, S4 further includes: Collect error logs for all failed test cases; The error log is converted into a high-dimensional vector using a text vectorization algorithm; Cluster analysis is performed on the high-dimensional vectors to automatically group errors with the same or similar root causes into one category; Generate a categorized summary report, showing the number of use cases and suspected root cause descriptions for each type of issue.
10. An automated interface testing system based on a constraint-aware large model, characterized in that, The system includes a natural language parsing module, a parameter constraint modeling module, a test case generation module, and a test execution and analysis module, wherein: The natural language parsing module is used to receive natural language test requirements input by testers, and to perform semantic analysis using a pre-trained language model that has been fine-tuned for the domain, breaking down the natural language test requirements into one or more test intentions. The parameter constraint modeling module is used to parse the corresponding standard interface documents, extract the constraint information of multiple parameters, and construct a parameter constraint matrix for one or more interfaces involved in test intents. The test case generation module is used to fuse the parameter constraint matrix and use it as input to the pre-trained language model that has been fine-tuned in the domain, and generate structured test cases that conform to the interface constraints through the slot filling strategy. The test execution and analysis module is used to dynamically convert the structured test cases into parameterized test scripts, execute them in the order of dependencies, collect execution logs, calculate test pass rate indicators, and generate visual test reports.
