Web interface automatic test method and system based on artificial intelligence

Abstract

The invention discloses a Web interface automatic testing method and system based on artificial intelligence, and the method mainly comprises the following steps: obtaining, comparing and summarizing diversified data based on a large language model, and outputting data, a metadata knowledge base and historical script data; and performing customized training on the large language model by adopting a fine tuning method, which comprises the steps of constructing training data, performing training by adopting a supervised fine tuning method, and performing continuous optimization based on feedback data. According to the method, the Web interface automatic test development efficiency can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of computer software testing technology, and in particular to an automated testing method and system for user interfaces (UI) based on artificial intelligence. Background Technology

[0002] Currently, automated testing of web interfaces typically employs the Page Object (PO) pattern, combined with tools such as Selenium and Robot Framework. However, with the frequent changes in web front-ends and the rapid pace of product iteration, existing methods generally suffer from the following shortcomings: 1. High degree of human involvement: Testers need to manually find page elements, maintain location information, and write metadata functions, which is inefficient, error-prone, and difficult to adapt to interface changes quickly.

[0003] 2. Insufficient metadata coverage: The richness and coverage of metadata are limited by the experience of testers, making it difficult to cope with complex scenarios.

[0004] 3. Poor automatic adaptability: When faced with frequent adjustments to the structure and attributes of Web elements, existing automation scripts are prone to failure and have high maintenance costs.

[0005] Artificial intelligence (AI) technology offers new solutions to the aforementioned problems, but current AI applications are often limited by issues such as scarce training data and poor model generalization ability, resulting in low accuracy in locating interface elements and insufficient usability of generated metadata and scripts. Therefore, how to systematically train and deploy AI remains a key challenge. Domain specialization Developing AI models and applying them efficiently in automated testing processes is key to improving the level of testing intelligence. Summary of the Invention

[0006] This disclosure provides an AI-based automated testing method and system for web interfaces, which significantly reduces repetitive tasks such as manually searching for page elements, maintaining locators, and writing metadata, improves the efficiency of automated test development, makes metadata generation more comprehensive and standardized, covers multiple test scenarios, facilitates automatic script conversion and maintenance, automatically adapts to changes in page structure, reduces the risk of automated test case failure, improves the robustness and scalability of testing, supports users to describe requirements in natural language, and lowers the technical threshold for automated testing.

[0007] This disclosure is based on the following approach: • After acquiring, comparing, and summarizing diverse data, output data, metadata knowledge base, and historical script data, and use fine-tuning methods to customize the training of the large language model.

[0008] • Training data construction: Training data is derived from diversified acquisition of page element structure (JSON format) output after comparison and induction, metadata definition and use case script. The data needs to be strictly cleaned, denoised and labeled, including manual verification of element visibility and interactivity, and accurate labeling of the mapping relationship between metadata and elements, to ensure the quality of training data.

[0009] • Training method: Supervised fine-tuning (SFT) method is used for training. The training data includes question and answer data of multiple tasks and terms in specific professional fields.

[0010] • Continuous optimization mechanism: A complete feedback data collection pipeline is established to automatically collect element recognition accuracy, script execution pass rate and other data. Model retraining triggers are set: when the accuracy drops to the preset index or enough new training data is received, the model optimization process is automatically started.

[0011] The artificial intelligence-based web interface automation testing method provided by the present disclosure mainly includes the following steps: S1, based on a large language model, after acquiring, comparing and inducing diversified data, output data, metadata knowledge base and historical script data; S2, fine-tuning method is used to customize training of large language model, including: Training data construction: Training data is derived from diversified acquisition of page element structure output in JSON format after comparison and induction, metadata definition and use case script; The method of supervised fine-tuning is used for training, and the training data includes question and answer data of multiple tasks and terms in specific professional fields; Based on feedback data, continuous optimization is performed.

[0012] Further, the step S1 specifically includes: S11, interface element automatic extraction: based on the DOM structure analysis method, the target web page information is structured extracted, and the AI model is combined to analyze the page DOM node attributes and hierarchical relationship, realizing the automatic identification and extraction of the interactive elements of the web interface. The identification result is output in structured JSON format, and is constrained through a self-defined data model to ensure data specification and subsequent processing universality; S12, metadata knowledge base construction: a structured and machine-readable test field knowledge base is constructed as the "memory" or "priori knowledge" of the large language model LLM, so that it has the ability to understand the specific concepts, relationships and logic of the test field; S13, script automatic conversion and output: the generated metadata function is parameter bound with user demand or structured test data, and the automatic test script file conforming to the syntax specification of the target test framework is generated through automatic filling and template assembly; the output script includes: test suite initialization, resource file reference, variable definition and metadata call, supports automatic parameter filling and special type format conversion, and generates the final executable automation test case file.

[0013] Further, the step S11 includes the following specific methods: Collecting real web page DOM samples of different types and structures; Sending each set of DOM structure data + unified structured input prompt to a large model without SFT to automatically identify the interactive elements of each page; Rule induction extraction, including: automatically analyzing the output of each group of the model to induce the recognition rules actually used by the model; counting the attribute distribution of the nodes recognized as interactive elements in each group of data; inducing the determination logic of the model in each group; recording the determination conditions of the elements in each output; Rule comparison and intersection feature generation: perform set operations on the determination rules induced in all groups to find the determination conditions commonly recognized by all groups, i.e., the intersection; Seed rule generation: use the intersection part as the "seed rule" to form the basic automatic determination standard; keep the unique rules of each group as supplements or abnormal features; Structured output of seed rules; Store the recognition results, rule extraction process, and intersection seed rules of each group of data in batches for subsequent SFT training and rule fine-tuning; Use the "seed rule" as the initial determination logic for subsequent SFT training to guide the construction of training samples and label generation.

[0014] Further, the step S12 includes the following specific methods for constructing the metadata knowledge base: (1) Element-metadata automatic mapping: Use a hierarchical decision mechanism to automatically match and generate corresponding metadata and parameter objects based on the unique identifier, interaction type, and attribute information of the interface element; among them, preferentially match specific element types and interactions (such as "input[type=text]+ can input"), and if the matching fails, automatically downgrade to a general rule (such as "elements with text attribute"), to ensure coverage and accuracy; (2) Parameter rule automatic extraction step; Multi-modal input (element attribute, annotation, historical case) collaborative modeling to dynamically generate KeywordMap and parameter objects; (3) Business process induction step, automatically extract the operation process between elements, form a structured business process map; (4) Structured knowledge base generation step; Confidence fusion: linear combination of interaction confidence, attribute completeness, and semantic relevance, set trainable weights (alpha, beta, and gamma), and realize reliable measurement of knowledge entries; High-confidence structured storage of the above information for large language model retrieval and reasoning.

[0015] Further, the step S13, the specific method of parameter automatic binding includes: Parameter induction table driving based on metadata type, design parameter binding decision table, automatically map different types of metadata (such as input, selection, and verification) and their required parameters. For example, input class automatically binds {text: d.value}, selection class automatically binds {option: d.value}, and verification class automatically binds {expected_text: d.value}. This regularized and table-driven parameter matching method greatly improves the accuracy and scalability of automatic script generation, facilitating the rapid support of new operation types in the future. Further, the step S2 specifically includes: Data uniform collection and preprocessing, the data including one or more of web page DOM structure, element and metadata mapping, business process description, and standard script; Structured Prompt and input-output design: design structured Prompt templates according to different tasks; Unified model training: based on the same large model backbone, use multi-task or multi-stage training method, and use supervised fine-tuning method for training; Model output standardization and acceptance; Continuous optimization and expansion: when new business requirements or data types are added, supplement data and Prompt templates to quickly adapt and expand model capabilities.

[0016] Further, the step of unified model training includes the following specific methods for model training of automatic interface element extraction: Data preparation; Diversified data collection: through automated tools, batch extract DOM data from different types and structures of web pages, covering diversified pages; Data formatting and cleaning; Rule seed generation: Collect multiple sets of web page DOM samples, each set containing different page types; input the original DOM data into the general un-tuned large model, prompt it to extract interactive elements; induce rules from the model output; compare multiple sets of rules, extract common conditions, and form a "seed rule set"; convert the seed rules into structured configurations; Training process design: input is the cleaned JSON format DOM data + seed rule set; output is the labeled training sample, containing one or more of element interaction type, locator, confidence; among them, use seed rules to automatically label part of the samples, and manually check the low confidence samples; Supervised fine-tuning: fine-tune the DeepSeek large language model based on the domain; Training task design: includes: interaction type classification, i.e. multi-classification task, used to learn to predict the interaction type of the element; locator generation, i.e. sequence generation task, used to generate unique identifiers according to priority; confidence score, i.e. regression task, output the confidence of the element recognition result; Rule and model fusion training: use seed rules as prior knowledge for model initialization, force the model to learn samples with clear rules in the early stage of training, gradually introduce complex scenarios, and enhance the generalization ability of the model.

[0017] Further, in the training data construction, the data needs to go through strict cleaning, denoising and labeling processing, including manual verification of element visibility and interactivity, and accurate labeling of the mapping relationship between metadata and elements, to ensure the quality of the training data.

[0018] Further, in the step S2, the specific method of continuous training optimization includes: Establish a complete feedback data collection pipeline to automatically collect element recognition accuracy and script execution pass rate data; Set up a model retraining trigger: when the accuracy drops below the preset index, or when new training data is received within the set threshold, automatically start the model optimization process. A web interface automation testing system based on artificial intelligence using the above method mainly includes: An interface element automatic extraction module is used to extract structured information from target web page information based on DOM structure analysis method, and analyze page DOM node attributes and hierarchical relationship combined with AI model, realize automatic identification and extraction of web interface interactive elements, and the identification result is output in structured JSON format, which is constrained by self-defined data model to ensure data specification and subsequent processing universality; A metadata knowledge base construction module is configured to construct a structured and machine-readable test field knowledge base as "memory" or "prior knowledge" of the large language model LLM, so that the LLM has the ability to understand test field specific concepts, relationships and logic. A script automatic conversion and output module is configured to bind the generated metadata function with user requirements or structured test data, fill in parameters automatically and assemble templates to generate an automatic test script file conforming to the syntax specification of a target test framework; the output script includes test suite initialization, resource file reference, variable definition and metadata calling, supports automatic parameter filling and special type format conversion, and generates a final executable automatic test case file. A model training module is configured to provide underlying model training and capability upgrade support for the three core business modules of interface element automatic extraction, metadata knowledge base construction and script automatic conversion and output; the module takes the page element structure, metadata definition and use case script in JSON format as training data, performs supervised fine-tuning training based on a large language model, and performs continuous training optimization based on feedback data.

[0019] Compared with the prior art, the present disclosure has the following advantages: ①Efficiency improvement: through the collaborative work of the three core modules, the whole process automation from page element recognition to test script generation is realized, eliminating the repetitive labor of traditional manual element searching, metadata writing and script maintenance; ②When the Web interface changes, the system can automatically re-identify elements and generate stable positioning identifiers based on priority, reducing the risk of script failure; ③Based on massive sample training and knowledge base driven mechanism, the system can automatically identify multiple interaction types and complex business scenarios, generate standardized test scripts, and effectively cover edge cases that are easily ignored by manual work; ④Users only need to describe test requirements in natural language, and the system can automatically complete semantic analysis, element mapping and script generation, so that non-technical personnel such as business experts can directly participate in automated testing; ⑤The "data collection-model training-deployment feedback" closed-loop optimization mechanism enables the system to continuously iterate and improve itself based on usage feedback. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the different views of the drawings.

[0021] Figure 1 The flowchart according to the exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Preferred embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure is more thoroughly and completely conveyed to those skilled in the art, and the scope of the present disclosure is fully conveyed to those skilled in the art.

[0023] The present disclosure provides an artificial intelligence-based Web interface automatic testing method and system.

[0024] The main steps of the method include: S1, based on a large language model, acquiring, comparing, and inducing diversified data, and outputting data, metadata knowledge base, and historical script data; S2, custom training of the large language model using a fine-tuning method, including: Training data construction: The training data is derived from the page element structure, metadata definition, and use case script output in JSON format after diversified acquisition, comparison, and induction; The training data contains question and answer data of multiple tasks and terms in specific professional fields; Based on feedback data, continuous optimization is performed. An exemplary system applying the method mainly includes an interface element automatic extraction module, a metadata knowledge base construction module, a script automatic conversion and output module, and a training module. The details are as follows: 1. Interface element automatic extraction module This module uses a DOM structure-based parsing method to structureally extract target web page information, and combines AI model analysis of page DOM node attributes and hierarchical relationships to achieve automatic recognition and extraction of web interface interactive elements. The recognition result is output in a structured JSON format, and is constrained by a custom data model to ensure data specification and subsequent processing versatility.

[0025] (1) Core functions DOM structure parsing and element recognition; diversified data acquisition and rule comparison and induction; structured JSON output.

[0026] (2) Diversified data acquisition and rule comparison and induction detailed design process: Step 1: Multiple sets of HTML / DOM data collection Source: Collect real web DOM samples of different types and structures.

[0027] Format: Each set of data is provided in HTML source code or structured JSON (DOM node) form.

[0028] Step 2: Large model automatic recognition (model without SFT) Input: Send each set of DOM structure data + unified structured input prompt (such as "please extract all interactive elements and their attribute descriptions") into the large model.

[0029] Output: The model returns the structured JSON of interactive elements for each page (including tag, attribute, interaction type, confidence, etc.).

[0030] Step 3: Rule induction extraction Automatically analyze each group of model output and induce the recognition rules actually used by the model: Statistical distribution of attributes (such as tag, type, attributes, etc.) of nodes recognized as interactive elements in each group of data; Induce the judgment logic of the model in each group (such as which attributes are considered "clickable"); Record the judgment conditions of elements in each group output (such as whether it contains onclick, role, type, etc.).

[0031] Step 4: Rule comparison and intersection feature generation Rule comparison: Perform set operations on all the judgment rules induced by each group to find the judgment conditions recognized by all groups (intersection). For example: <button>、 If it has onclick and role=button, it is clickable; <input type="text"> Input is allowed.

[0032] Seed rule generation: The intersection is used as "seed rules" to form the basic automated judgment criteria; the unique rules of each group are retained as supplementary or abnormal features.

[0033] Step 5: Structured output of seed rules Based on the module output schema, organize the seed rule set (such as decision functions, attribute weights, classification criteria, etc.) and output structured JSON / configuration file.

[0034] Step 6: Dataset and Rule Storage The identification results, rule extraction process, and intersection seed rules for each set of data are stored in batches for subsequent SFT training, rule fine-tuning, etc.

[0035] Step 7: Provide a foundation for subsequent fine-tuning By using "seed rules" as the initial decision logic for subsequent SFT training, training sample construction and label generation are guided, thereby improving training efficiency and effectiveness.

[0036] (3) Explanation of the core algorithm ① Rule induction (single sample) From the response (Rᵢ) of a large model, we can summarize which judgment rules (Fᵢ) it used.

[0037] Formula: Fᵢ = Φ(Rᵢ) Explanation: Φ stands for "rule extractor," which is responsible for inferring rules from results.

[0038] ② Rule comparison (multiple samples) Compare the rules summarized from all samples and find the "consensus rule" (C) that everyone agrees on.

[0039] Formula: C = Ψ(F1, F2, ..., F n ) Explanation: Ψ is a "rule filter," and its core operation is to find the intersection (∩), that is, C = F1 ∩ F2 ∩ ... ∩ F n Only rules common to all samples are retained.

[0040] ③ Complete process The entire algorithm is a pipeline from data to the final rule: General formula: S=Ω(Ψ(Φ(LLM(D1)),…,Φ(LLM(Dn)))) Interpretation: Large Model View (LLM): Look at a webpage and tell me which parts are clickable.

[0041] We guess (Φ): Based on its answer, guess what its rule is for determining "clickable".

[0042] We look for commonalities (Ψ): compare all the rules it guesses and find the one it uses every time.

[0043] We write it down (Ω): We write this universally accepted rule in a way that machines can understand.

[0044] IV. Input / Output Format Input: Web page DOM structure (HTML / JSON format) Output: Structured element information (JSON Schema validation) { "List of elements": [ { "tag": "input", "id": "username", "type": "text", "xpath": " / / input[@id='username']", "unique_id": "id=username", "interaction_type": "Enterable", "confidence": 0.95 } ] } 2. Metadata Knowledge Base Construction Module The core objective of this module is to build a structured, machine-readable test domain knowledge base as the "memory" or "prior knowledge" of the Large Language Model (LLM), enabling it to understand test domain-specific concepts, relationships, and logic.

[0045] (1) Core functions Automatic element-metadata mapping; automatic extraction of parameter rules; business process summarization; generation of structured knowledge base.

[0046] (2) Explanation of the core algorithm ① Element-metadata mapping: The KeywordMap(e) mapping relationship is shown in the table below: Conditions Keywords e.interaction_type = "clickable" and e.tag = "button" click_button e.interaction_type = "clickable" and e.tag = "a" click_link e.interaction_type = "input" and e.type = "text" input_text e.interaction_type = "input" and e.type = "password" input_password e.interaction_type = "select" and e.tag = "select" select_from_dropdown e.interaction_type = "select" and e.type = "checkbox" toggle_checkbox e.interaction_type = "clickable" click_element e.interaction_type = "input" input_text e.interaction_type = "select" select_element Others verify_element_present illustrate: e represents an element object.

[0047] interaction_type represents the interaction type of the element (such as "clickable", "inputable", "optionable"); tag, type represent the HTML tag name and type attribute; Priority is determined from top to bottom, and if the condition is met, the corresponding metadata is returned.

[0048] ② Parameter rule automatic extraction Explanation: e: element object KeywordMap(e): metadata obtained by element mapping e.unique_id: unique identification of the element e.text: element text content e.alt: alt attribute of the picture {...}: represents the generated parameter object ∅: represents no parameter generation ③ Confidence calculation function: Confidence(e)=α⋅interaction_confidence+β⋅attribute_completeness+γ⋅semantic_relevance Where: α+β+γ=1 (weight coefficient) interaction_confidence: interaction type recognition confidence, based on model recognition / rule hit degree quantification (such as 1.0 for complete hit, 0.8 for partial) attribute_completeness: element attribute completeness score. The completeness of the required key attributes of the element.

[0049] The calculation formula is attribute_completeness = actual number of key attributes possessed / total number of key attributes required An input element requires {id, type, placeholder, label}, and actually has {id, type, placeholder}, so attribute_completeness = 3 / 4 = 0.75 semantic_relevance: semantic relevance score based on text, alt, context, etc. Similarity First, identify the keywords and their synonyms for the target metadata (e.g., input_text, click_element, etc., which could correspond to "input", "click", "submit", "login", etc.). Second, determine whether the semantic attributes of the element contain the target operation keyword or its synonyms. Finally, a match will result in a high score; otherwise, a low score will be awarded.

[0050] Final mapping output: MappingResult(e)=⟨KeywordMap(e),Params(e),Confidence(e)> The algorithm structure adopts a hierarchical decision-making mechanism, which prioritizes matching specific element types and interaction combinations, and gradually degrades to general mapping rules to ensure that appropriate test metadata and parameters can be generated in various scenarios.

[0051] (3) Input / output format Input: Receive structured JSON (containing unique identifiers, interaction types, attributes, confidence levels, etc.) output by the UI element auto-scraping module, as well as metadata definition files, function comments, and historical test scripts.

[0052] Output: { Page Name: Login Page Module: "User Management" Description: "User authentication entry page", "List of elements": [ { "unique_id": "id=username", "tag": "input", "type": "text", "interaction_type": "Enterable", "Metadata operations": [ { "keyword": "input_text", "Parameters": { "locator": "id=username", "text": "username" }, Instructions: Enter username } ], Confidence level: 1.0 }, { "unique_id": "id=submit-btn", "tag": "button", "interaction_type": "clickable", "metadata_operations": [ { "keyword": "click_element", "parameters": { "locator": "id=submit-btn" }, "description": "Click the login button" } ], "confidence": 1.0 } ], "common_flows": [ "input_text: id=username -> input_text: id=password -> click_element:id=submit-btn" ], "low_confidence_tips": []} 3. Script automatic conversion and output module This module binds the generated metadata function with user requirements or structured test data through parameter binding, automatic filling and template assembly, and generates automated test script files that meet the syntax specifications of target test frameworks (such as Robot Framework).

[0053] The output script includes test suite initialization, resource file reference, variable definition and metadata call, supports automatic parameter filling and special type format conversion, and generates the final executable automated test case file.

[0054] (1) Core functions Parameter automatic binding; executable file generation.

[0055] (2) Core algorithm description: Parameter binding algorithm: BindParams(k, d) = k.params∪ {text: d.value} if k.keyword∈ INPUT_KEYWORDS k.params ∪ {option: d.value} else if k.keyword ∈ SELECT_KEYWORDS k.params ∪ {expected_text: d.value} elseif k.keyword ∈ VERIFY_KEYWORDS k.params otherwise Conditions Binding parameters Explanation k.key ∈ INPUT_KEYWORDS k.params ∪ {text: d.value} Input-type keywords (e.g., input) k.keyword ∈ SELECT_KEYWORDS k.params ∪ {option: d.value} Select-type keywords (e.g., select) k.keyword ∈ VERIFY_KEYWORDS k.params ∪ {expected_text: d.value} Verify-type keywords (e.g., verify) Others k.params No additional binding Chinese description k: metadata object, contains metadata (k.keyword) and existing parameters (k.params).

[0056] d: data object, contains value to be bound (d.value).

[0057] INPUT_KEYWORDS: set of input-type metadata (e.g. input_text, input_password).

[0058] SELECT_KEYWORDS: set of select-type metadata (e.g. select_from_dropdown).

[0059] VERIFY_KEYWORDS: set of verify-type metadata (e.g. verify_text).

[0060] ∪: set union operation, means adding new parameters to existing parameters.

[0061] k.params: original parameters.

[0062] {text: d.value}, {option: d.value}, {expected_text: d.value}: parameters to be appended when binding different types of metadata.

[0063] RobotFramework statement formatting function: FormatRFStatement(b) = b.keyword ++ JoinParams(b.params) if simple statement Indent(b.keyword ++ JoinParams(b.params)) if block statement Chinese description: •b.keyword: Metadata of the statement (such as Click Element).

[0064] • b.params: The set of parameters for the statement.

[0065] •JoinParams(b.params): Joins a set of parameters into a parameter string, using spaces or a specified delimiter.

[0066] •Indent(...): Adds indentation to the generated statements (e.g., for use within a code block).

[0067] • Simple statement: A simple statement that does not require indentation.

[0068] • Block statement: Block statements (such as those inside loops and conditional blocks) require indentation.

[0069] (3) Input / output format ① Input content: Structured knowledge base entries (such as JSON data of element-metadata-parameters) User requirements / business scenario description (natural language or structured test cases) Processing logic: • Parse user requirements or use case descriptions to extract the business processes and parameters to be tested.

[0070] • Match the corresponding metadata operations and parameter rules in the knowledge base.

[0071] • Automatically completes parameter filling (such as variable names, default values, format conversions, etc.).

[0072] Example: Structured knowledge base entries: { "element": "Frm_NtpServer1", "operation": "input_text", "parameters": { "locator": "id=Frm_NtpServer1", text: "" }, "description": "Enter the NTP server address"} • User input requirements: Configure the NTP server address as ntp.example.com • Parameter binding result: input_text id=Frm_NtpServer1 ntp.example.com ②Template assembly and script splicing Automatic splicing: Insert the metadata call with bound parameters into the corresponding position of the template; • Hold the flow control (such as pre and post steps), data-driven, loop and other extended syntax.

[0073] Example: • Robot Framework template snippet: *** Settings *** Resource resource.robot *** Variables *** ${NTP_ADDR} ntp.example.com *** Test Cases *** Configure NTP server input_text id=Frm_NtpServer1 ${NTP_ADDR} click_element id=Btn_Apply ③Format conversion and syntax checking Automatic formatting: adjust indentation, separators, variable formats, etc. according to the requirements of the target framework; Syntax checking: automatically detect the structure and syntax of the script, and automatically prompt or revert to manual correction if there are errors.

[0074] ④Output and integration File output: generate.robot,.py, etc. format files that meet the framework specifications.

[0075] Resource reference: automatically insert resource files, public metadata, etc. reference.

[0076] Executable verification: optionally integrate an automatic run to ensure script executability.

[0077] 4、Model training module This module, as a deep integration of underlying supporting technology, is deeply integrated into the three modules of the above interface element automatic extraction module, metadata knowledge base construction module and script automatic conversion and output module, and is the core of the system intelligence.

[0078] (1) Core functions As a unified intelligent engine, this training module is responsible for providing underlying model training and capability upgrade support for the three core business modules of "interface element automatic extraction", "metadata knowledge base construction", and "script automatic conversion and output". Based on large language models (such as DeepSeek), this module is supervised fine-tuned, and through structured input, unified output format, and efficient data closed loop, it realizes knowledge transfer and capability sharing in multiple scenarios.

[0079] (2) Core process ① Unified data collection and preprocessing Summarize the training samples of the three business modules, including web DOM structure, element and metadata mapping, business process description, standard script, etc. Clean, label and structure according to the unified data Schema, ensure that the data of each business sub-task can meet the unified input and output requirements of large models; Add security filtering (such as XSS detection), format checking and other preprocessing; ② Structured Prompt and input / output design According to different tasks (element extraction, intelligent web element and semantic mapping, script generation), design multiple but unified style structured Prompt templates; The input is unified as system role description, context information, business description, task instruction and original data segment; The output is machine-readable, structured JSON or script text, suitable for subsequent automated processing.

[0080] ③ Unified model training Based on the same large model backbone, use multi-task or multi-stage training method to support the fine-tuning of three types of core tasks; Tasks can reuse features, semantic understanding, and structured generation capabilities to achieve knowledge generalization; Support continuous incremental training (such as dynamic supplement of manually reviewed and corrected data).

[0081] ④ Model output standardization and acceptance All model outputs strictly follow the agreed Schema format, facilitating interface calls and subsequent process chaining; Integrate format checking, field integrity checking, and exception result marking for automatic acceptance; Support manual review and feedback loop for low confidence samples.

[0082] ⑤ Continuous optimization and extension When new business needs or data types are added, supplement data and Prompt templates to quickly adapt and extend model capabilities.

[0083] (3) Unified input and output of training modules ① Input Structured task samples: including DOM structure, element attributes, metadata definition, business process, script template, etc. Task type identification: indicates which module task the current training sample belongs to; Task instructions and constraints: such as "please identify all interactive elements", "please assign test metadata to elements", "please generate automated scripts", etc. ② Output Standardized JSON or scripts: such as element information, metadata knowledge entries, Robot Framework test code, etc. Confidence score and exception flag; Format verification results or manual review prompts.

[0084] (4) Detailed training methods based on modules ① Interface element automatic extraction module Data preparation: Data preparation: Diversified data collection: Through automation tools such as Selenium and Playwright, batch capture DOM data from different types and structures of web pages, covering common pages such as network security products and various pages to ensure sample richness.

[0085] Data formatting and cleaning: ▪ All collected DOM structures are converted to structured JSON or standard HTML format according to a unified Schema.

[0086] ▪ Automatically filter hidden, invalid, script-generated nodes that have no actual interactive significance to improve data effectiveness.

[0087] Rule seed generation: Generate initial rule seeds (used to guide model training) through the following process: ▪ Step 1: Collect multiple sets of web page DOM samples (each set contains different page types).

[0088] ▪ Step 2: Input the original DOM data into a general large model (not fine-tuned) and prompt it to extract interactive elements (for example: "Please extract all interactive elements and their attributes").

[0089] ▪ Step 3: Induce rules from model output (such as counting the labels and attribute distribution of recognized elements).

[0090] ▪ Step 4: Compare multiple rules and extract common conditions (such as all groups believe that < / button> <button> 、 To make it clickable), forming a "seed rule set".

[0091] Step 5: Convert seed rules to structured configuration (e.g., decision functions, attribute weights in JSON format).

[0092] Training process: Training data construction: Input: Cleaned DOM data (JSON format) + seed rule set.

[0093] Output: Labeled training samples (including element interaction type, locator, confidence).

[0094] Annotation method: Use seed rules to automatically annotate part of the samples (e.g., label < / button> <button>Element labels marked as "clickable" are marked as "clickable"). Manually check low-confidence samples to ensure the accuracy of the labels.

[0095] Supervised Fine-Tuning (SFT): Model Selection: Fine-tune large language models such as DeepSeek for domain-specific fine-tuning.

[0096] Training Task Design: Task 1: Interaction Type Classification (Multi-Classification Task): Learn to predict the interaction type of elements (clickable, inputable, selectable, etc.).

[0097] Task 2: Identifier Generation (Sequence Generation Task): Generate unique identifiers according to priority (e.g., data-testid > id > name > composite XPath).

[0098] Task 3: Confidence Scoring (Regression Task): Output the confidence of element recognition results (0~1).

[0099] Rule and Model Fusion Training Use seed rules as prior knowledge for model initialization, for example: ▪ Force the model to preferentially learn samples with clear rules in the early stages of training (e.g. <input type="text"> Must be inputable).

[0100] ▪ Gradually introduce complex scenarios (such as dynamically generated elements) to enhance model generalization ability.

[0101] ② Meta Data Knowledge Base Construction Module Data Preparation: Source: Element Extraction Module Output, Meta Data Definition File, Historical Test Scripts Labeling: Element-Meta Data Mapping Relationship, Parameter Rules, Business Processes Cleaning: Standardize attribute names, remove invalid mappings Training Process: Data Input and Preprocessing; Three-task joint training: Element-Meta Data Matching Task; Parameter Extraction and Induction Task; Knowledge Base Article Generation Task; Output Format Verification; Difficult Example Recycling and Optimization.

[0102] ③ Script Automatic Conversion and Output Module Data Preparation: Source: Knowledge Base Entries, Test Requirement Descriptions, Historical Script Templates Labeling: Parameter Binding Relationships, Template Selection Rules, Syntax Constraints Cleaning: Standardize script format, remove syntax errors samples Training process: Data annotation Prompt engineering / fine-tuning (recommended to prioritize Prompt engineering) Continuous optimization.

[0103] The main innovation in this embodiment is: • Domain-specific AI model construction Innovatively proposes a method for training data construction and model fine-tuning in the field of UI testing, through multi-task joint training to obtain a specialized model that deeply understands the interactive semantics of Web elements.

[0104] • Knowledge internalization intelligent mapping mechanism Internalize structured test knowledge through model fine-tuning into the underlying capabilities of AI models, achieving more accurate and smooth metadata generation and script conversion, which is different from traditional external knowledge base retrieval solutions.

[0105] • Fusion of rules and AI recognition algorithms Proposes an interactive element comprehensive scoring model that combines multi-dimensional attribute features and AI semantic prediction scores, taking into account the explainability of rules and the flexibility of AI, significantly improving the accuracy of element recognition on complex pages.

[0106] • End-to-end fully automatic pipeline For the first time, seamlessly integrates three major modules: element recognition, knowledge base construction, and script conversion, achieving "one-click generation" from user requirements to executable scripts, breaking through the limitations of traditional tool flow fragmentation.

[0107] • Intelligent closed-loop optimization system Designs a complete performance monitoring and retraining triggering mechanism that automatically starts the optimization process when the accuracy decreases or the number of new samples reaches a certain value, ensuring that the system continuously maintains technological advancement.

[0108] The content of this embodiment is different from the prior art: • Fundamental difference from traditional recording and playback tools: Traditional tools (such as Selenium IDE) are based on coordinates or static XPath recording operations, which are extremely fragile. This embodiment is based on specialized AI models for semantic-level element understanding and positioning, generating identifiers (such as data-testid) that are more stable and can dynamically adapt to page changes, fundamentally solving the high maintenance cost pain point of recording and playback tools.

[0109] • The core difference from existing AI-based testing tools: some current solutions only call general large model APIs for simple question answering or code generation, lacking deep field adaptation. This embodiment emphasizes end-to-end self-owned field model training, deployment and data closed loop. More importantly, it internalizes professional knowledge into model capability through fine-tuning, rather than relying only on external knowledge base retrieval. This "endogenous intelligence" solution has significant advantages in response speed, accuracy and business fit.

[0110] • Difference from pure rule engine method: pure rule method (such as fixed priority-based locator generation) is difficult to deal with complex and variable real-world scenarios. This embodiment combines rules (such as element identifier generation priority) with AI models, handling ambiguous and complex situations that rules cannot cover, achieving the complementary advantages of rule efficiency and AI intelligence.

[0111] The above technical solutions are only exemplary embodiments of the present application. For those skilled in the art, on the basis of the application disclosed herein, various types of improvements or modifications can be easily made without being limited to the methods described in the above embodiments. Therefore, the above-described method is only preferred and not limited.< / button>

Claims

1. An artificial intelligence-based web interface automation testing method, characterized by, Comprise the following steps: S1, based on large language model, after obtaining, comparing, inducing diversified data, output data, metadata knowledge base and historical script data; S2, using fine-tuning method to customize training of large language model, including: Training data construction: training data comes from page element structure, metadata definition and use case script output in JSON format after diversified acquisition, comparison and induction; Adopting supervised fine-tuning method for training, the training data contains multiple task question and answer data and specific professional terms; Based on feedback data, continuous optimization is carried out.

2. The method of claim 1, wherein, The step S1 specifically comprises: S11, interface element automatic capture: based on DOM structure analysis method, the information of target web page is structured extracted, and the AI model is combined to analyze the page DOM node attribute and hierarchical relationship, the automatic identification and extraction of web page interface interactive elements are realized, the identification result is output in structured JSON format, and the data is constrained through self-defined data model to ensure data specification and subsequent processing universality; S12, metadata knowledge base construction: construct structured and machine-readable test field knowledge base as "memory" or "priori knowledge" of large language model LLM, so that it has the ability to understand specific concepts, relationships and logic in test field; S13, script automatic conversion and output: the generated metadata function is parameter bound with user demand or structured test data, the automatic test script file conforming to the syntax specification of target test framework is generated through automatic filling and template assembly; The output script includes: test suite initialization, resource file reference, variable definition and metadata call, supports automatic parameter filling and special type format conversion, and generates the final executable automation test case file.

3. The method of claim 2, wherein, The specific method of step S11 comprises: Collect real web page DOM samples of different types and structures; Send each set of DOM structure data + unified structured input prompt to the large model without SFT, and automatically identify the interactive elements of each page; Rule induction and extraction, including: automatically analyzing each group output of the model, and inducing the recognition rules actually used by the model; Statistics of the attribute distribution of the nodes recognized as interactive elements in each group of data; Induce the judgment logic of the model in each group; Record the judgment conditions of the elements in each output; Rule comparison and intersection feature generation: perform set operation on the judgment rules induced by all groups to find the judgment conditions recognized by all groups, that is, the intersection; Seed rule generation: take the intersection part as "seed rule" to form the basic automatic judgment standard; Retain the unique rules of each group as supplements or abnormal features; Seed rule structured output; Store the recognition results, rule extraction process and intersection seed rule of each group of data in batches for subsequent SFT training and rule fine-tuning; Use "seed rule" as the initial judgment logic for subsequent SFT training to guide the training sample construction and label generation.

4. The method of claim 2, wherein, The specific method of step S12, metadata knowledge base construction, comprises: (1) Element-metadata automatic mapping: Adopting a hierarchical decision mechanism, based on the unique identification of interface elements, interaction types, and attribute information, automatically matching corresponding metadata and parameter objects; Among them, specific element types and interactions are matched first, and if the matching fails, it is automatically downgraded to general rules to ensure coverage and accuracy; (2) Automatic extraction of parameter rules: Multi-modal input collaborative modeling, dynamically generating KeywordMap and parameter objects; (3) Business process induction: Automatically extract the operation process between elements to form a structured business process map; (4) Structured knowledge base generation: Confidence fusion: Linearly weight the interaction recognition confidence, attribute completeness, and semantic relevance, set trainable weights, and realize reliable measurement of knowledge items; High-confidence structured storage of the above information for large language model retrieval and reasoning.

5. The method of claim 2, wherein, The specific method of parameter automatic binding in step S13 includes: Parameter induction table driving based on metadata type, designing parameter binding decision table, and automatically mapping different types of metadata and their required parameters.

6. The method of claim 1, wherein, The step S2 specifically includes: Data collection and preprocessing, including one or more of web page DOM structure, element and metadata mapping, business process description, and standard script; Structured Prompt and input / output design: Design structured Prompt templates according to different tasks; Unified model training: Based on the same large model backbone, use multi-task or multi-stage training method, and use supervised fine-tuning method for training; Model output standardization and acceptance; Continuous optimization and expansion: When new business requirements or data types are added, supplement data and Prompt templates to quickly adapt and expand model capabilities.

7. The method of claim 6, wherein, The step of the unified model training, the specific method of model training for automatically grabbing interface elements includes: Data preparation; Diversified data collection: Use automated tools to batch grab DOM data from web pages of different types and structures, covering a variety of pages; Data formatting and cleaning; Rule seed generation: Collect multiple sets of web page DOM samples, each containing different page types; input the original DOM data into the general un-tuned large model, prompt it to extract interactive elements; summarize rules from model output; compare multiple rules to extract common conditions and form a "seed rule set"; Convert the seed rule into a structured configuration; Training process design: Input is the cleaned JSON format DOM data + seed rule set; output is the labeled training sample, including one or more of element interaction type, locator, and confidence; among them, use seed rules to automatically label part of the samples, and manually check low-confidence samples; Supervised fine-tuning: Fine-tune the DeepSeek large language model based on the domain; Training task design: including: interaction type classification, i.e. multi-classification task, used to learn to predict the interaction type of elements; locator generation, i.e. sequence generation task, used to generate unique identifiers according to priority; confidence score, i.e. regression task, output the confidence of element recognition results; Rule and model fusion training: Seed rules are used as prior knowledge for model initialization. In the early stage of training, the model is forced to learn samples with clear rules first, and gradually introduces complex scenarios to enhance the model's generalization ability.

8. The method of claim 1, wherein, During training data construction, data needs to undergo strict cleaning, denoising and labeling processing, including manual verification of element visibility and interactivity, and accurate labeling of the mapping relationship between metadata and elements to ensure the quality of training data.

9. The method of claim 1, wherein, The specific method of continuous training optimization in step S2 includes: Establish a complete feedback data collection pipeline to automatically collect element recognition accuracy and script execution pass rate data; Set up a model retraining trigger: when the accuracy drops below the preset index or receives new training data below the set threshold, automatically start the model optimization process.

10. An artificial intelligence-based web interface automation testing system applying the method of any one of claims 1-9, characterized by, It includes: Interface element automatic extraction module, which is used to extract the information of target web page based on DOM structure analysis method, and combines AI model to analyze page DOM node attributes and hierarchical relationship, realizes automatic identification and extraction of web interface interactive elements, and the identification result is output in structured JSON format, which is constrained by custom data model to ensure data specification and subsequent processing universality; Metadata knowledge base construction module, used to construct structured and machine-readable test field knowledge base as "memory" or "prior knowledge" of large language model LLM, so that it has the ability to understand test field specific concepts, relationships and logic; Script automatic conversion and output module, used to bind the generated metadata function with user demand or structured test data, fill in automatically and assemble templates to generate automatic test script files that meet the syntax specifications of the target test framework; Output script includes: test suite initialization, resource file reference, variable definition and metadata call, supports automatic parameter filling and special type format conversion, and generates the final executable automated test case file; Model training module, used to provide underlying model training and capability upgrade support for interface element automatic extraction, metadata knowledge base construction, script automatic conversion and output three core business modules; This module uses JSON format output of page element structure, metadata definition and test script as training data, based on large language model for supervised fine-tuning training, and based on feedback data, for continuous training optimization.

Images