Mixed-mode test case to visual automation script conversion method and system

Abstract

The application discloses a mixed-state test case to visual automation script conversion method and system, and belongs to the technical field of software test automation. The method comprises the following steps: constructing a UI-Data knowledge graph; receiving a mixed-state test case, classifying the intention of single test content in the test case, and identifying at least one of three types of intention labels, namely, a UI operation type, a visual assertion type and a hidden state type; based on the knowledge graph and the test content identified as the hidden state type, converting the backend hidden state into a front-end observable visual feature assertion description; generating a structured test decision sequence according to at least the visual feature assertion description, the content identified as the UI operation type and the content identified as the visual assertion type; converting the structured test decision sequence into a framework executable API calling instruction through a target visual automation framework, and generating an executable visual automation test script. The scheme can realize accurate identification and classification of test step intentions.

Description

Technical Field

[0001] This application belongs to the field of software test automation technology, and specifically relates to a method and system for converting hybrid test cases into visual automation scripts. Background Technology

[0002] Currently, in enterprise software development processes, manual test cases are generally written in natural language, and their core characteristic is mixed-state description: test steps not only include explicit UI interactive operations such as "clicking a button" and "entering text," but also a large number of non-interactive backend implicit state verification requirements such as "checking the database order status to change to paid" and "verifying that the interface returns normal user permissions." According to the RIPR defect detection model, the effective detection of a software defect must simultaneously satisfy four conditions: reachability, error state generation, error state propagation, and observability. However, existing automated testing technologies generally have shortcomings: traditional DOM automation tools can only solve the reachability problem and cannot track the propagation of backend error states; visual regression testing tools can only implement UI explicit feature assertions and cannot handle backend implicit state verification; general large model code generation can only achieve static syntax transformation and cannot understand business state transitions and cascading side effects.

[0003] Existing automated testing technologies cannot fully address the end-to-end automated transformation requirements of hybrid test cases. Their core shortcomings fall into four categories: First, the inherent defects of DOM-based automation technologies, including selector fragility (frequent changes in the DOM structure lead to over 67% test failures) and state blindness (unable to understand business state transitions and completely lacking the ability to handle implicit backend state validation); second, the limitations of visual regression testing technologies, which are essentially "assertions" rather than "executors," unable to read natural language test cases to automatically generate operation sequences, and completely unable to handle backend states such as databases and interfaces; third, the dynamic blindness problem of general large model-assisted generation technologies, where models can only understand static syntax and cannot predict cascading side effects caused by operations, resulting in insufficient script coverage and poor robustness; and fourth, the decision gap in existing visual automation frameworks. While new-generation frameworks, such as Midscene.js, have solved the problems of "how to execute" and "how to verify," they cannot automatically deduce frontend visual verification features when faced with implicit state validation steps, leaving a core decision gap of "what to verify."

[0004] In summary, existing technologies cannot fully realize the end-to-end automatic conversion of hybrid test cases into highly robust visual automation scripts. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method and system for converting hybrid test cases into visual automation scripts.

[0006] This application provides a method for converting hybrid test cases into visual automation scripts, including: A UI-Data knowledge graph is constructed by using at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relationship extraction. The UI-Data knowledge graph is used to describe the mapping relationship between UI elements and back-end data states. The test cases are received in a hybrid state and described in natural language. The BERT-LLM hybrid two-layer architecture is used to classify the intent of each test item in the test cases and identify at least one of the three intent labels: UI operation, visual assertion, and implicit state. For test content identified as latent state, entity recognition is performed to extract the business entities contained therein, the business attributes corresponding to the business entities, and the target state values ​​that need to be verified for the business attributes. Using the extracted business entities and the target state values ​​as query conditions, a reverse state propagation path search is performed in the UI-Data knowledge graph to track UI elements that have a direct or indirect relationship with the latent state, and a candidate visual feature set is generated based on the searched UI elements. The scores of each visual feature in the candidate visual feature set are calculated under multiple evaluation indicators, including visual saliency, feature confidence, detection stability, and historical execution success rate. The comprehensive score of each visual feature is calculated based on the scores of each evaluation indicator, and the comprehensive scores are sorted from high to low. A preset number of visual features with the highest comprehensive scores are selected, and corresponding visual feature assertion descriptions are generated based on the selected visual features. Based on at least the visual feature assertion description, the test content identified as UI operation class, and the test content identified as visual assertion class, a platform-independent structured test decision sequence is generated; the structured test decision sequence is converted into API call instructions executable by the target visual automation framework through the target visual automation framework, and combined to generate an end-to-end executable visual automation test script.

[0007] In this embodiment of the application, the construction of the UI-Data knowledge graph further includes: During the execution of the test script, runtime feedback information is collected. Based on the runtime feedback information, the UI-Data knowledge graph is updated with feature confidence and the mapping relationship is supplemented, thereby realizing the iterative evolution of the UI-Data knowledge graph.

[0008] In this embodiment, a BERT-LLM hybrid two-layer architecture is used to classify the intent of individual test content in the test cases, identifying at least one of three intent tags: UI operation, visual assertion, and implicit state. Specifically, this includes: The Sentence-BERT model is invoked to map a single test content into a semantic embedding vector. The cosine similarity between the semantic embedding vector and three predefined intent center vectors is calculated to obtain three similarity scores. The three intent center vectors include: the first center vector corresponding to the UI operation class, the second center vector corresponding to the visual assertion class, and the third center vector corresponding to the latent state class. Select the maximum similarity score from the three similarity scores. If the maximum similarity score is greater than or equal to the preset feature confidence threshold, output the intent label corresponding to the maximum similarity score. If the maximum similarity score is less than the preset feature confidence threshold, then construct a few-sample prompt word containing the thought chain, call the large language model to perform intent reasoning on a single test content, and output the intent label returned by the large language model. Among them, UI operation intent tags are used to represent test content that needs to simulate user interaction behavior, visual assertion intent tags are used to represent test content that needs to verify the state or content of front-end UI elements, and implicit state intent tags are used to represent test content that needs to verify the state of back-end data and that the state has no direct corresponding UI element.

[0009] In this embodiment of the application, visual features are sorted from high to low based on their comprehensive scores, and a preset number of visual features with the highest comprehensive scores are selected. Based on the selected visual features, corresponding visual assertion descriptions are generated. Prior to this, the process also includes: Determine whether the candidate visual feature set is empty or whether the comprehensive score is lower than a preset threshold; If the candidate visual feature set is empty, generate API verification code as an alternative verification scheme, and output a warning message indicating that there is no front-end UI performance in this state, as well as suggestions for improving observability. If the candidate visual feature set is not empty but the overall score is lower than the preset threshold, a hybrid verification scheme of visual assertion and API verification is generated, and manual review prompts and observability improvement suggestions are output for suggesting the addition of UI features or reviewing the verification results.

[0010] In this embodiment of the application, at least based on visual feature assertion descriptions, test content identified as UI operation classes, and test content identified as visual assertion classes, a platform-independent structured test decision sequence is generated, including: At least the UI operation test content, visual assertion content, and visual feature assertion description should be structured and arranged according to the test execution logic to generate a platform-independent structured test decision sequence that includes test step identifiers, original test content, intent labels, operation instructions, assertion instructions, and execution order information.

[0011] In this embodiment of the application, the method, after converting the structured test decision sequence into API call instructions executable by the target visual automation framework and combining them to generate an end-to-end executable visual automation test script, further includes: The target visual automation framework executes the interactive operations corresponding to the UI operation classes and / or the assertion verification corresponding to the visual feature assertion descriptions in the visual automation test scripts.

[0012] In this embodiment of the application, the interaction operations corresponding to the UI operation classes and / or assertion verifications corresponding to the visual feature assertion descriptions in the visual automation test script are executed through the target visual automation framework, including: Get a screenshot of the current interface, and generate a mapping between interactive elements and icons based on the screenshot; Based on the correspondence and the natural language description of UI operation classes and / or visual feature assertion classes, determine the target element; perform corresponding interactive operations and / or assertion verification based on the target element.

[0013] In this embodiment of the application, the method constructs a UI-Data knowledge graph through at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relationship extraction. Prior to this, it also includes: Establish L1 coordinate cache, L2 visual label cache, L3 inference result cache, and L4 visual language model response cache; During the execution of the visual automated test script, the corresponding level cache is queried first. If the cache is hit, it is directly reused and the calculation step is skipped. If the cache is not hit, the calculation is performed and the result is written to the cache. Monitor the execution process of visual automated test scripts, capture automated test script execution failure information; identify and clear cache items at all levels associated with failure information, and adjust at least one of the following based on the failure information: VLM call parameters, element location threshold, assertion wait timeout, or number of retries; Based on the failure information, the feature confidence value of the corresponding mapping relationship in the UI-Data knowledge graph is reduced, and the failure mode is recorded as a negative sample. Based on the updated UI-Data knowledge graph and the adjusted parameters, the structured test decision sequence is regenerated and recombined to generate an executable visual automation test script.

[0014] This application also provides a hybrid test case to visual automation script conversion system, including: The graph construction module is used to construct a UI-Data knowledge graph through at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relationship extraction. The UI-Data knowledge graph is used to describe the mapping relationship between UI elements and back-end data states. The intent classification module is used to receive mixed-state test cases described in natural language. It adopts a BERT-LLM hybrid two-layer architecture to classify the intent of a single test content in the test case and identify at least one of the three types of intent tags: UI operation, visual assertion, and implicit state. The causal reasoning module is used to perform entity recognition on test content identified as latent state classes, extracting the business entities contained therein, the business attributes corresponding to the business entities, and the target state values ​​that need to be verified for the business attributes. Using the extracted business entities and target state values ​​as query conditions, it performs a reverse state propagation path search in the UI-Data knowledge graph to track UI elements that have direct or indirect relationships with latent states, and generates a candidate visual feature set based on the searched UI elements. It calculates the score of each visual feature in the candidate visual feature set under multiple evaluation indicators, including visual saliency, feature confidence, detection stability, and historical execution success rate. It calculates the comprehensive score of each visual feature based on the scores of each evaluation indicator, sorts them from high to low comprehensive scores, selects a preset number of visual features with the highest comprehensive scores, and generates corresponding visual feature assertion descriptions based on the selected visual features. The script generation module is used to generate a platform-independent structured test decision sequence based at least on visual feature assertion descriptions, content identified as UI operation classes, and content identified as visual assertion classes. The structured test decision sequence is then converted into API call instructions executable by the target visual automation framework and combined to generate an end-to-end executable visual automation test script.

[0015] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the hybrid test case to visual automation script conversion method provided in the above embodiments.

[0016] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the hybrid test case to visual automation script conversion method provided in the above embodiments.

[0017] The solution proposed in this application can resolve the semantic gap between mixed-state test cases described in natural language and executable automated code, and achieve accurate identification and classification of the intent of test steps.

[0018] This application can also solve the problem that implicit backend states cannot be directly verified through frontend UI automation, thus filling the decision gap of "what to verify" in existing visual automation frameworks.

[0019] This application also completely eliminates the reliance on DOM selectors, adopting a pure visual positioning solution to resolve the issue of high-frequency script failures caused by UI changes. Furthermore, this application enables the fully automated conversion of hybrid test cases from natural language text to directly executable visual automation scripts, without requiring manual intervention to supplement verification points and operation instructions.

[0020] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a method for converting hybrid test cases into visual automation scripts according to an embodiment of this application is shown.

[0023] Figure 2 A flowchart illustrating the automatic construction process of a UI-Data knowledge graph according to an embodiment of this application is shown.

[0024] Figure 3 A diagram illustrating the collaborative relationship between embodiments of this application and existing visual automation frameworks is shown.

[0025] Figure 4 A flowchart illustrating the execution of the BERT-LLM hybrid two-layer intent classification algorithm according to an embodiment of this application is shown.

[0026] Figure 5 A schematic diagram of the causal inference engine workflow according to an embodiment of this application is shown.

[0027] Figure 6 A flowchart of the Set-of-Mark visual positioning technology according to an embodiment of this application is shown.

[0028] Figure 7 A before-and-after comparison of SoM markings according to an embodiment of this application is shown.

[0029] Figure 8 A schematic diagram illustrating the entire process of scenario transformation according to an embodiment of this application is shown.

[0030] Figure 9A general architecture diagram of a hybrid test case to visual automation script conversion system according to an embodiment of this application is shown. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] Terminology Explanation Hybrid test cases: These are manually written test cases in natural language that simultaneously include explicit UI interaction steps and implicit verification steps for backend states. This type of test case is the mainstream form in enterprise software testing, emphasizing the integration of frontend operations and backend state verification.

[0033] Implicit state: refers to the system backend state that cannot be directly observed through the front-end UI, including database field values, API return data, server-side cached content, and results of background scheduled tasks. Its verification usually relies on indirect methods rather than visual feedback from the interface.

[0034] Reverse Oracle: The core mechanism of this application refers to the reasoning process of deriving a set of visual features that can be stably observed on the front-end UI from a known back-end target state. This mechanism aims to solve the problem of visually verifying implicit states, enabling the back-end state to be indirectly verified through front-end behavior.

[0035] UI-Data Knowledge Graph: The domain knowledge graph constructed in this application is used to structurally store the triggering relationships, display relationships, and dependencies among UI components, business data entities, and system states. This graph constitutes the data foundation for causal reasoning mechanisms, supporting the mapping and deduction between implicit states and UI performance.

[0036] Set-of-Mark (SoM) visual positioning: A UI element positioning technology based on Visual Model (VLM). This technology overlays numerical markers onto interactive elements in a screenshot, guiding the VLM to accurately locate target elements based on visual information, thus completely replacing the traditional DOM selector positioning method.

[0037] Figure 1 A flowchart illustrating the method for converting hybrid test cases into visual automation scripts according to an embodiment of this application is shown, such as... Figure 1 As shown, the method includes the following steps: S1. Construct a UI-Data knowledge graph using at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relationship extraction. The UI-Data knowledge graph is used to describe the mapping relationship between UI elements and back-end data states.

[0038] It's worth mentioning that UI structure extraction involves extracting existing UI elements and hierarchical relationships from the front-end interface layout and control tree (such as the DOM for web and the View tree for mobile devices); API documentation parsing involves analyzing state binding and conditional rendering logic in front-end code (such as Vue / React); front-end code static analysis involves analyzing state binding and conditional rendering logic in front-end code (such as Vue / React); and large language model relationship extraction involves automatically extracting the "state → performance" relationship from requirements documents, test cases, and design documents using LLM.

[0039] This application constructs an initial UI-Data knowledge graph using at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relation extraction. At this point, the initial UI-Data knowledge graph includes at least: back-end state nodes, front-end visual feature nodes, state-feature mapping relationships, feature confidence scores for each relationship, and the source of the relationship.

[0040] S2. Receive mixed-state test cases described in natural language, and use the BERT-LLM hybrid two-layer architecture to classify the intent of individual test content in the test cases, and identify at least one of the three types of intent labels: UI operation, visual assertion, and implicit state.

[0041] S3. For test content identified as latent state, perform entity recognition, extract the business entities contained therein, the business attributes corresponding to the business entities, and the target state values ​​that need to be verified for the business attributes; use the extracted business entities and the target state values ​​as query conditions, perform a reverse state propagation path search in the UI-Data knowledge graph, track UI elements that have a direct or indirect relationship with the latent state, and generate a candidate visual feature set based on the searched UI elements; calculate the score of each visual feature in the candidate visual feature set under multiple evaluation indicators, including visual saliency, feature confidence, detection stability, and historical execution success rate; calculate the comprehensive score of each visual feature based on the scores of each evaluation indicator, sort them from high to low comprehensive scores, select a preset number of visual features with the highest comprehensive scores, and generate corresponding visual feature assertion descriptions based on the selected visual features.

[0042] S4. Based on at least the visual feature assertion description, the test content identified as UI operation class, and the test content identified as visual assertion class, generate a platform-independent structured test decision sequence; convert the structured test decision sequence into API call instructions executable by the target visual automation framework through the target visual automation framework, and combine them to generate an end-to-end executable visual automation test script.

[0043] Furthermore, the aforementioned "structured test decision sequence" refers to: the test content obtained after intent classification is structured and arranged according to test execution logic, generating an ordered sequence containing at least the following elements: Test step identifier, used to uniquely identify each test step; Original test content, used to preserve the original information described in natural language; Intent labels are used to identify the category of the step, including at least one of the following: UI action, visual assertion, and implicit state. Operation commands are used to describe the UI interaction actions that need to be performed; Assertion directives are used to describe the front-end visual features or states that need to be verified. Execution order information is used to indicate the order or dependencies between steps; The structured test decision sequence is a platform-independent intermediate representation that is provided as input to the target visual automation framework to generate specific executable visual automation test scripts.

[0044] This application addresses the semantic gap between hybrid test cases described in natural language and executable automation code, thereby enabling accurate identification and classification of test step intent. Furthermore, it solves the problem that implicit backend states cannot be directly verified through frontend UI automation, thus filling the decision-making gap in existing visual automation frameworks regarding "what to verify."

[0045] Furthermore, this application completely eliminates the reliance on DOM selectors in existing technologies. By adopting a pure visual positioning scheme, it can solve the problem of high-frequency script failures caused by UI changes. Simultaneously, this application can achieve fully automated conversion of hybrid test cases from natural language text to directly executable visual automation scripts, without requiring manual intervention to supplement verification points and operation instructions.

[0046] The collaboration boundary between this application and existing visual automation frameworks is shown in Table 1 below: Table 1

[0047] As shown in Table 1, the core difference between this application and the prior art is that the prior art only solves the "how" problem (execution layer), while this application adds the "what" problem (decision layer) on this basis. Through causal reasoning, knowledge graphs and intent classification, the implicit state of the backend is automatically converted into a visual test script that can be executed by the frontend, and the existing framework is reused to complete the execution, thereby realizing a complete closed loop from "natural language test cases" to "end-to-end automated test scripts".

[0048] Figure 2 A flowchart illustrating the automatic construction process of a UI-Data knowledge graph according to an embodiment of this application is shown. Figure 3 A diagram illustrating the collaborative relationship between embodiments of this application and existing visual automation frameworks is shown, such as... Figure 2 and Figure 3 As shown, based on the above embodiments, the construction of the UI-Data knowledge graph in this application further includes: collecting runtime feedback information during the execution of the test script, updating the feature confidence and supplementing the association relationships of the UI-Data knowledge graph according to the runtime feedback information, so as to realize the iterative evolution of the UI-Data knowledge graph.

[0049] For example, in the initial knowledge graph, the confidence score of (Order.status=Paid, triggers, UI.Label_OrderStatus.Text="Paid") is 0.98. After 100 test script executions, this assertion succeeded 98 times and failed 2 times. Based on the historical success rate, the system updates the feature confidence score to 0.98 (keeping it high). However, the assertion "Refund button visible" only succeeded 72 times, and the feature confidence score dropped from 0.96 to 0.69. In subsequent inferences, this feature is downgraded and recommended.

[0050] In the initial knowledge graph, Order.status=Paid only associated with the order status tag and did not record the "Payment Successful" Toast notification. After the test script was executed 5 times consecutively, the feedback self-healing engine found that a "Payment Successful, Points +100" Toast notification appeared every time payment was successful. The system automatically added a new triplet (Order.status=Paid, triggers, UI.Toast_Payment Successful.Visible=true), and the initial feature confidence was set to 0.85.

[0051] This application dynamically adjusts feature confidence, giving higher recommendation priority to high-frequency success paths; by supplementing through mapping relationships, missing UI features are automatically entered into the graph, and a more complete set of visual assertions can be generated during subsequent implicit state transitions, thus achieving continuous optimization of the UI-Data knowledge graph.

[0052] In summary, the UI-Data knowledge graph in this application adopts a complete closed-loop mechanism of "building—using—updating": During the construction phase, the system pre-establishes a mapping relationship between backend state and frontend visual features through at least one of the following methods: UI structure extraction, API documentation parsing, frontend code static analysis, and large language model relationship extraction, forming an initial UI-Data knowledge graph.

[0053] During the usage phase, when the system identifies implicit state-type test content, it calls the initial UI-Data knowledge graph and uses a reverse oracle causal reasoning algorithm to convert the backend implicit state into an observable visual feature assertion description, which is then used to generate a structured test decision sequence.

[0054] During the update phase, the system continuously collects runtime feedback information during the execution of the test script, including execution success / failure results, visual assertion hit status, location failure information, rollback mechanism trigger records, and manual correction results.

[0055] Based on the above feedback, this application can dynamically adjust the feature confidence of mapping relationships in the UI-Data knowledge graph: relationships with stable hits increase feature confidence, while relationships with frequent failures decrease feature confidence.

[0056] Meanwhile, this application supplements the knowledge graph with associations based on newly discovered state-performance relationships and manual correction results (as incremental learning samples), realizing the iterative evolution of the UI-Data knowledge graph and making it more accurate and complete in subsequent reasoning.

[0057] This application is based on the UI-Data knowledge graph. ,in: Node set It includes three types of core entities: UI component nodes: interactive / displayable front-end elements such as Button, Label, Input, and Table; Data entity nodes: business data objects such as Order, User, and Product; Business status nodes: Paid, Closed, Active, Shipped, and other business enumeration states.

[0058] edge set It includes three core relationships, and each edge is accompanied by a feature confidence weight (0-1): Displays: Data entities / states are displayed through UI components; Triggers: State changes trigger changes to the style, text, and visibility of UI components; depends_on: Business dependencies between entities / states.

[0059] Based on the above embodiments, the three-stage process for automated knowledge graph construction is as follows: First, the multi-source data automatic crawling stage: UI structure extraction: traversing the core business processes of the application under test through a headless browser, automatically extracting the DOM tree, component hierarchy, CSS properties, text content, and interaction events; API document parsing: automatically parsing Swagger / OpenAPI specification documents, extracting business entity definitions, field types, state enumeration values, and interface dependencies; Static code analysis: performing AST abstract syntax tree parsing on the front-end source code, identifying the mapping relationship between state management logic such as Redux / Vuex and UI rendering.

[0060] Second, the LLM-driven relation extraction stage: Construct a small sample Prompt, input the UI component descriptions, API field definitions, and code contexts crawled from multiple sources into the LLM, and automatically extract the <entity, relation, entity> triples.

[0061] Third, the manual verification and incremental learning stage: After automatically generating candidate maps, the system supports lightweight review and correction by testing experts; the system automatically records the manually corrected content as a few-sample example, and automatically adapts to similar scenarios in the future; during runtime, the system continuously collects execution feedback data, automatically optimizes the feature confidence of high-frequency paths, and marks low-frequency paths as pending review, so as to realize the continuous iteration of the map.

[0062] Figure 4 The following is a flowchart illustrating the execution of the BERT-LLM hybrid two-layer intent classification algorithm according to an embodiment of this application: Figure 4 As shown in this embodiment, a BERT-LLM hybrid two-layer architecture is adopted to classify the intent of a single test item in the test case, identifying at least one of three intent tags: UI operation, visual assertion, and implicit state. This includes the following steps: S4. Call the Sentence-BERT model to map a single test content into a semantic embedding vector, and calculate the cosine similarity between the semantic embedding vector and the three predefined intent center vectors to obtain three similarity scores. The three intent center vectors include: the first center vector corresponding to the UI operation class, the second center vector corresponding to the visual assertion class, and the third center vector corresponding to the latent state class.

[0063] For example, in a specific instance, the input to the Sentence-BERT model is a single test step text. (Including operation description and expected results), the output is a step intent label. ,in: UI interaction operation steps, corresponding to physical interaction actions; Visual assertion steps, corresponding to the inspection of explicit UI features; Implicit state-based steps, corresponding to non-visual validation methods such as database / API checks.

[0064] This application uses a pre-trained Sentence-BERT model (all-mpnet-base-v2) to process the input test step text. Mapped to a fixed 768-dimensional semantic embedding vector .

[0065] Predefined set of center vectors for three types of intentions Each center vector is calculated from the mean of the labeled samples of the corresponding category.

[0066] The embedding vector is calculated using the cosine similarity formula. degree of matching with each center vector:

[0067] in, Let L2 norm be the vector. , The center vector of the predefined j-th type of intent.

[0068] Set the confidence threshold for classification features. (Default value is 0.85, which can be fine-tuned according to business scenarios). If the maximum similarity score satisfies:

[0069] Then output directly The corresponding intent label is used to complete the classification; otherwise, the second-level LLM fine-tuning process is initiated.

[0070] For fuzzy steps where BERT cannot classify with high feature confidence, a few-sample Prompt containing a CoT (Cooperation of Thought) is constructed, and LLM is invoked for precise intent inference. The core structure of the Prompt can be as follows: [Task Description] You need to categorize the intent of the software testing steps, and only three types of labels can be output: UiAct, VisAssert, and HideState. The categorization rules are as follows: 1. UiAct: Includes the user's physical interaction with the UI, such as clicking, inputting, swiping, selecting, etc. 2. VisAssert: Only includes checks and validations of visible features of the front-end UI, such as checking text display, button status, page navigation, etc. 3. HideState: Contains validation of states that are not visible to the backend, such as database fields, API return data, cache status, and server-side task execution results.

[0071]

Example

[0072]

enter

[0073] Output After LLM outputs the category labels, the entire intent classification process is completed.

[0074] This application utilizes a reverse oracle mechanism to automatically convert backend implicit states into frontend visual assertions. Its mathematical essence lies in solving for the posterior probability. The maximization problem, i.e., given the hidden state Solve for the set of visual observation features that are most likely to occur and can be reliably detected. .

[0075] S5. Select the maximum similarity score from the three similarity scores. If the maximum similarity score is greater than or equal to the preset feature confidence threshold, output the intent label corresponding to the maximum similarity score.

[0076] S6. If the maximum similarity score is less than the preset feature confidence threshold, construct a few-sample prompt word containing the thought chain, call the large language model to perform intent reasoning on a single test content, and output the intent label returned by the large language model.

[0077] Among them, UI operation intent tags are used to represent test content that needs to simulate user interaction behavior, visual assertion intent tags are used to represent test content that needs to verify the state or content of front-end UI elements, and implicit state intent tags are used to represent test content that needs to verify the state of back-end data and that the state has no direct corresponding UI element.

[0078] In one example, this application employs Causal Chain Inference to automatically convert the back-end hidden states into observable visual feature assertions for the front end. The input to this algorithm is the hidden state description. (e.g., "Order.Status = Paid"), the output of this algorithm is a set of visual assertion descriptions. Alternatively, a fallback flag and alternative validation scheme can be used.

[0079] As shown in Table 2: Table 2

[0080] Based on the above embodiments, the core weights in the causal chain derivation algorithm are calculated using the following standardized method: Visual salience weight This is used to assess the prominence of visual features in a UI, and the calculation formula is as follows: .

[0081] in: The area ratio of visual features on the screen, normalized to [0,1]; The color contrast between visual features and the background is calculated based on the WCAG standard and normalized to [0,1]. : Centrality of visual features, 1 for the center of the screen and decreasing for the edge regions, normalized to [0,1].

[0082] Detectable stability weights Used to evaluate whether visual features can be stably detected by VLM, the calculation formula is: .

[0083] in: Animation coefficient: 1 if the feature includes animation / gradient effects, otherwise 0; Transient coefficient, which is 1 when the element is a temporary display element such as a hover tooltip or Toast, and 0 otherwise.

[0084] Overall score Calculation formula: The final comprehensive score of candidate visual features is calculated using a weighted product method to avoid feature invalidation due to excessively low weights for individual features.

[0085] in, For feature confidence weights, Weighted by historical execution success rate. These are adjustable hyperparameters, with a default value of 1.0. The weight and priority can be adjusted according to the business scenario.

[0086] like Figure 5 As shown in this embodiment, visual features are sorted from high to low based on their comprehensive scores, and a preset number of visual features with the highest comprehensive scores are selected. A corresponding set of visual assertion descriptions is generated based on the selected visual features. Prior to this, the process also includes: S7. Determine whether the candidate visual feature set is empty or whether the comprehensive score is lower than the preset threshold.

[0087] S8. If the candidate visual feature set is empty, generate API verification code as an alternative verification scheme, and output a warning message indicating that there is no front-end UI performance in this state, as well as suggestions for observability improvement.

[0088] S9. If the candidate visual feature set is not empty but the overall score is lower than the preset threshold, a hybrid verification scheme of visual assertion and API verification is generated, and manual review prompts and observability improvement suggestions are output for suggesting the addition of UI features or reviewing the verification results.

[0089] In this embodiment of the application, at least based on the visual feature assertion description, the test content identified as UI operation class, and the test content identified as visual assertion class, a platform-independent structured test decision sequence is generated, including: at least the test content of UI operation class, the test content of visual assertion class, and the visual feature assertion description are arranged in a structured manner according to the test execution logic to generate a platform-independent structured test decision sequence containing test step identifiers, original test content, intent labels, operation class instructions, assertion class instructions, and execution order information.

[0090] Figure 6 A flowchart of the Set-of-Mark visual positioning technology according to an embodiment of this application is shown. Figure 7 The image comparison diagram before and after the SoM mark according to the embodiment of this application is shown, such as Figure 6 and Figure 7 As shown, the hybrid test case to visual automation script conversion method provided in this application converts the structured test decision sequence into API call instructions executable by the target visual automation framework, and combines them to generate an end-to-end executable visual automation test script. After that, it also includes: executing the interactive operations corresponding to the UI operation class in the visual automation test script and / or the assertion verification corresponding to the visual feature assertion description through the target visual automation framework.

[0091] Furthermore, the target visual automation framework executes the interaction operations corresponding to the UI operation classes and / or assertion verifications corresponding to the visual feature assertion descriptions in the visual automation test scripts, including: S10. Obtain a screenshot of the current interface and generate a correspondence between interactive elements and icons based on the screenshot.

[0092] Specifically, the process involves capturing a screenshot of the current screen, detecting interactive elements in the screenshot, and overlaying unique numerical tags onto the detected interactive elements to generate a tagged image.

[0093] S11. Determine the target element based on the correspondence and the natural language description of the UI operation class and / or visual feature assertion class; perform the corresponding interactive operation and / or assertion verification based on the target element.

[0094] Specifically, the labeled image and the natural language operation or assertion description to be performed are input into the visual language model. The visual language model recognizes and returns the numerical tag corresponding to the target element. Based on the numerical tag, the target element in the interface is located, and the corresponding interactive operation or assertion verification is performed.

[0095] For example, in a specific instance, this application uses SoM visual positioning technology, combined with existing visual automation frameworks, to achieve highly accurate and low-latency UI interaction and assertion execution, thus solving the brittleness problem of DOM selectors.

[0096] The execution steps of the Set-of-Mark (SoM) visual localization algorithm include: Step 1: Take a screenshot of the application being tested. ; Step 2: Call the lightweight object detection model DINOv2 to identify all interactive UI elements in the screenshot and generate bounding boxes for each element. ;in, The x-coordinate (in pixels) of the top-left corner of the bounding box; : The y-coordinate (in pixels) of the top-left corner of the bounding box; : The x-coordinate (in pixels) of the bottom right corner of the bounding box; : The y-coordinate (in pixels) of the bottom right corner of the bounding box.

[0097] Step 3: On the original screenshot, overlay a semi-transparent number label onto the bounding box of each interactive element to generate a labeled image. ; Step 4: Construct a Prompt, The input VLM is the target of the operation. Example Prompt: "I need to click the [Submit] button. Please tell me the number of the numeric label corresponding to the button. Just output the number." Step 5: VLM returns the numeric label corresponding to the target element. The system matches the bounding box corresponding to the label and calculates the center coordinates of the element. ; Step 6: Call the underlying browser driver to simulate mouse clicks / text input and other operations to complete the UI interaction.

[0098] Furthermore, the hybrid test case to visual automation script conversion method provided in this application constructs a UI-Data knowledge graph through at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relationship extraction. Previously, it also included: S12. Establish L1 coordinate cache, L2 visual marker cache, L3 inference result cache, and L4 visual language model response cache. Before performing localization, inference, or model invocation, first query the corresponding level cache. If a match is found, the cache is reused directly and the calculation step is skipped. If no match is found, the calculation is performed and the result is written to the cache.

[0099] This application addresses the issues of high inference latency and high token consumption in VLM by designing a four-level Cache-First caching architecture. The core design is shown in Table 3 below: Table 3

[0100] As shown in Table 3, this application adopts the LRU eviction algorithm and sets a cache expiration period. When execution fails, the corresponding cache entry is automatically invalidated, and inference and location are re-executed to ensure the accuracy of the cache.

[0101] S13. Monitor the execution process of the visual automated test script, capture the failure information of the automated test script execution; identify and clear the cache items at all levels associated with the failure information, and adjust at least one of the following according to the failure information: VLM call parameters, element location threshold, assertion wait timeout time or number of retries.

[0102] S14. Based on the failure information, reduce the feature confidence value of the corresponding mapping relationship in the UI-Data knowledge graph and record the failure mode as a negative sample; based on the updated UI-Data knowledge graph and the adjusted parameters, regenerate the structured test decision sequence and recombine it to generate an executable visual automation test script.

[0103] Figure 8 This paper illustrates a schematic diagram of the entire process transformation of an embodiment according to the present application. A test scenario of an e-commerce platform order payment process is presented below. Figure 8 The following is an explanation based on specific examples: I. Implementation Scenarios and Prerequisites: System under test: Web interface of a B2C e-commerce platform; Test objective: Verify the functional correctness of the entire order payment process; Prerequisites: Automated construction and manual review of the e-commerce platform's UI-Data knowledge graph have been completed; The target execution framework is Midscene.js, and the environment has been deployed and adapted; The test account is logged in, and there are pending orders in the shopping cart with order number ORD20260224001; Intent classification threshold. Visual salience threshold Maximum propagation depth .

[0104] As shown in Table 4: Table 4

[0105] II. Original Mixed-State Manual Test Cases: Test Case ID: TC-ORDER-001; Test Case Name: Full-Link Verification of Order Payment Process; Preconditions: The user has logged into the system, has orders to be paid in the shopping cart, and has sufficient account balance.

[0106] III. System end-to-end execution flow: Input layer parsing: The system receives test cases in Excel format, automatically parses them into 4 structured test steps, and extracts the operation description and expected result of each step.

[0107] Intent-based execution: Steps 1-2: After BERT embedding calculation, the similarity with the UiAct center vector is 0.92 and 0.94 respectively, both ≥0.85, and it is directly classified as UiAct (UI operation class). Steps 3-4: After BERT embedding calculation, the similarity with the center vector of HideState is 0.81 and 0.78 respectively, both <0.85, so it enters the LLM fine-tuning process and is finally classified as HideState (latent state class).

[0108] Causal reasoning engine execution (step 3): Entity recognition: The entity is parsed as Order, the attribute is status, and the target value is Paid; Graph retrieval: Found 3 edges associated with Order.status: (Order.status, displays, UI.Label_Order Status), with a feature confidence level of 0.98; (Order.status=Paid, triggers, UI.Button_ImmediatePay.Visibility=false), feature confidence level is 0.95; (Order.status=Paid, triggers, UI.Button_RequestRefund.Visibility=true), feature confidence level is 0.96; State propagation simulation: propagation depth is 0, no cascading side effects, and 3 candidate visual features are generated; Weight calculation and score ranking: The comprehensive scores of the three candidate features are 0.94, 0.89 and 0.91, respectively, all ≥0.6; Output the top-3 visual assertion set: The order status label on the order details page displays "Paid"; The "Pay Now" button is not visible on the order details page; The "Request a Refund" button is visible on the order details page.

[0109] Causal reasoning engine execution (step 4): Entity recognition: Parse out the deduction record with entity "Inventory", attribute "record", and target value "order ORD20260224001"; Graph retrieval: No displays or triggers related to Inventory.record were found, and no candidate visual features were found. Trigger rollback mechanism: Generate an API_VERIFICATION type rollback solution, automatically generate inventory interface verification code, and output a warning: "This inventory deduction record has no front-end UI behavior. API verification code has been automatically generated. It is recommended to add corresponding UI features to improve testability."

[0110] Script generation and framework adaptation: The system converts UI operation commands, visual assertions, and API validation code into TypeScript scripts executable by the Midscene.js framework. The complete script is as follows: Execution layer operation and feedback: The script executes automatically, and all UI operations and assertions are completed through the VLM capabilities of Midscene.js, without the need for DOM selectors. After execution, a test report is generated, and all steps have a 100% pass rate. The execution results are fed back to the system, updating the feature confidence of the corresponding cache and UI-Data knowledge graph.

[0111] The following is a specific example illustrating the implementation of a scheduled task verification scenario (Fallback mechanism implementation) in a backend management system: I. Implementation Scenarios and Prerequisites: System under test: E-commerce platform merchant backend management system; Test objective: Verify the correctness of the scheduled task for automatically archiving expired coupons in the backend; Prerequisites: The UI-Data knowledge graph of the backend management system has been completed, and the target execution framework is Playwright + AI plugin.

[0112] II. Core steps of the original test cases: Steps: After triggering the expired coupon archiving scheduled task, verify that the expired coupon cache in Redis has been cleared, and that coupons with an expired field of 1 in the coupon table in the database have been archived to the coupon_history table.

[0113] III. Implementation Process and Results: Intent Classification: This step is classified as a HideState implicit state step; Causal Reasoning: After querying the UI-Data knowledge graph, no front-end UI display features corresponding to Redis cache state and coupon table archived data were found, and no candidate visual features were found; Rollback Mechanism Trigger: A HYBRID hybrid verification scheme was generated. On the one hand, API verification code for Redis and the database was automatically generated. On the other hand, a manual prompt was output: "The execution result of this scheduled task has no front-end UI display. Interface and database verification code has been generated. It is recommended to add a scheduled task execution result display panel on the management side to improve observability"; Script Generation: A Playwright automation script containing database query and Redis state verification was automatically generated, which can be directly executed to complete the test verification.

[0114] IV. Performance Optimization Implementation Methods: In this embodiment, the system adopts a four-level caching architecture and a high-concurrency scheduling scheme to achieve performance optimization: Cache preheating: Before test execution, based on the test case sequence, page SoM tagging, causal reasoning, and coordinate calculation are performed in advance to complete L1-L3 cache preheating; Asynchronous batch processing: Combines multiple aiAssert assertions on the same page into a single VLM call, reduces API round trips through multiple issue prompts, and reduces token consumption by 60%; Edge computing deployment: Deploy lightweight VLM (LLaVA-7B) and DINOv2 models locally to handle simple element localization and assertions, while only uploading complex causal reasoning and semantic understanding to the cloud-based large model, reducing end-to-end latency by 70%; Parallel execution: Through a multi-process parallel execution engine, 50 test cases can be run simultaneously on a single node, improving execution efficiency by more than 40 times.

[0115] Final performance metrics: First execution (no cache): average 5-8 seconds / step, including the entire process of graph query, VLM inference, and causal inference; Cache hit scenario: average 0.5-1 seconds / step, latency reduced by more than 90%; High concurrency scenario: single node supports 50+ test cases to be executed in parallel without performance bottlenecks.

[0116] This application also provides a hybrid test case to visual automation script conversion system, including: a graph construction module, an intent classification module, a causal reasoning module, and a script generation module.

[0117] The graph construction module is used to construct a UI-Data knowledge graph through at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relationship extraction. The UI-Data knowledge graph is used to describe the mapping relationship between UI elements and back-end data states.

[0118] The intent classification module is used to receive mixed-state test cases described in natural language. It adopts a BERT-LLM hybrid two-layer architecture to classify the intent of individual test content in the test cases and identify at least one of the three types of intent tags: UI operation, visual assertion, and implicit state.

[0119] The causal reasoning module is used to perform entity recognition on test content identified as latent state classes, extracting the business entities contained therein, the business attributes corresponding to the business entities, and the target state values ​​that need to be verified for the business attributes. Using the extracted business entities and target state values ​​as query conditions, it performs a reverse state propagation path search in the UI-Data knowledge graph to track UI elements that have direct or indirect relationships with latent states, and generates a candidate visual feature set based on the searched UI elements. It calculates the score of each visual feature in the candidate visual feature set under multiple evaluation indicators, including visual saliency, feature confidence, detection stability, and historical execution success rate. It calculates the comprehensive score of each visual feature based on the scores of each evaluation indicator, sorts them from high to low comprehensive scores, selects a preset number of visual features with the highest comprehensive scores, and generates corresponding visual feature assertion descriptions based on the selected visual features.

[0120] The script generation module is used to generate a platform-independent structured test decision sequence based at least on visual feature assertion descriptions, content identified as UI operation classes, and content identified as visual assertion classes. The structured test decision sequence is then converted into API call instructions executable by the target visual automation framework and combined to generate an end-to-end executable visual automation test script.

[0121] This application addresses the semantic gap between hybrid test cases described in natural language and executable automation code, thereby enabling accurate identification and classification of test step intent. Furthermore, it solves the problem that implicit backend states cannot be directly verified through frontend UI automation, thus filling the decision-making gap in existing visual automation frameworks regarding "what to verify."

[0122] Furthermore, this application completely eliminates the reliance on DOM selectors in existing technologies. By adopting a pure visual positioning scheme, it can solve the problem of high-frequency script failures caused by UI changes. Simultaneously, this application can achieve fully automated conversion of hybrid test cases from natural language text to directly executable visual automation scripts, without requiring manual intervention to supplement verification points and operation instructions.

[0123] Figure 9 The overall architecture diagram of the hybrid test case to visual automation script conversion system according to an embodiment of this application is shown, such as... Figure 9 As shown in Table 5 below, the input definitions for the hybrid test case to visual automation script conversion system in this application embodiment are as follows: Table 5

[0124] Furthermore, the output definition of the hybrid test case to visual automation script conversion system in this application embodiment is shown in Table 6 below: Table 6

[0125] The operating environment of the hybrid test case to visual automation script conversion system of this application embodiment is shown in Table 7 below: Table 7

[0126] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the hybrid test case to visual automation script conversion method provided in the above embodiments.

[0127] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the hybrid test case to visual automation script conversion method provided in the above embodiments.

[0128] The hardware architecture of electronic devices / devices can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. A bus connects various circuits, including one or more processors, memories, and / or hardware modules. A bus can also connect various other circuits such as peripherals, voltage regulators, power management circuits, external antennas, etc. Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Component (EISA) buses, etc. Buses can be categorized as address buses, data buses, control buses, etc.

[0129] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0130] The specific implementation of each module in the above-mentioned device can be referred to the implementation process of the corresponding steps in the above-mentioned method implementation method of this disclosure, and will not be repeated here.

[0131] This application also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of readable storage media include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0132] This application also provides a computer program product, the methods of which can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0133] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for converting hybrid test cases into visual automation scripts, characterized in that, include: A UI-Data knowledge graph is constructed by at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relationship extraction. The UI-Data knowledge graph is used to describe the mapping relationship between UI elements and back-end data states. The test cases are received in a hybrid state and described in natural language. The BERT-LLM hybrid two-layer architecture is used to classify the intent of each test item in the test cases and identify at least one of the three intent labels: UI operation, visual assertion, and implicit state. For test content identified as latent state, entity recognition is performed to extract the business entities contained therein, the business attributes corresponding to the business entities, and the target state values ​​that need to be verified for the business attributes. Using the extracted business entities and the target state values ​​as query conditions, a reverse state propagation path search is performed in the UI-Data knowledge graph to track UI elements that have a direct or indirect relationship with the latent state, and a candidate visual feature set is generated based on the searched UI elements. The scores of each visual feature in the candidate visual feature set are calculated under multiple evaluation indicators, including visual saliency, feature confidence, detection stability, and historical execution success rate. A comprehensive score for each visual feature is calculated based on the scores of each evaluation indicator, and the comprehensive scores are sorted from high to low. A preset number of visual features with the highest comprehensive scores are selected, and corresponding visual feature assertion descriptions are generated based on the selected visual features. Based at least the visual feature assertion description, the test content identified as the UI operation class, and the test content identified as the visual assertion class, a platform-independent structured test decision sequence is generated; the structured test decision sequence is converted into API call instructions executable by the target visual automation framework through the target visual automation framework, and combined to generate an end-to-end executable visual automation test script.

2. The method according to claim 1, characterized in that, The construction of the UI-Data knowledge graph also includes: During the execution of the test script, running feedback information is collected. Based on the running feedback information, the feature confidence of the UI-Data knowledge graph is updated and the mapping relationship is supplemented, thereby realizing the iterative evolution of the UI-Data knowledge graph.

3. The method according to claim 1, characterized in that, A BERT-LLM hybrid two-layer architecture is used to classify the intent of individual test content in test cases, identifying at least one of three intent tags: UI operation, visual assertion, and implicit state. Specifically, these include: The Sentence-BERT model is invoked to map a single test content into a semantic embedding vector. The cosine similarity between the semantic embedding vector and three predefined intent center vectors is calculated to obtain three similarity scores. The three intent center vectors include: the first center vector corresponding to the UI operation class, the second center vector corresponding to the visual assertion class, and the third center vector corresponding to the latent state class. Select the maximum similarity score from the three similarity scores. If the maximum similarity score is greater than or equal to a preset feature confidence threshold, then output the intent label corresponding to the maximum similarity score. If the maximum similarity score is less than the preset feature confidence threshold, then a few-sample prompt word containing the thought chain is constructed, the large language model is called to perform intent reasoning on a single test content, and the intent label returned by the large language model is output. Among them, UI operation intent tags are used to represent test content that needs to simulate user interaction behavior, visual assertion intent tags are used to represent test content that needs to verify the state or content of front-end UI elements, and implicit state intent tags are used to represent test content that needs to verify the state of back-end data and that the state has no direct corresponding UI element.

4. The method according to claim 1, characterized in that, The visual features are sorted from highest to lowest based on their overall scores. A predetermined number of visual features with the highest overall scores are selected. Based on these selected visual features, corresponding visual assertion descriptions are generated. Previously, this also included: Determine whether the candidate visual feature set is empty or whether the comprehensive score is lower than a preset threshold. If the candidate visual feature set is empty, an API verification code is generated as an alternative verification scheme, and a warning message indicating that there is no front-end UI performance in this state and suggestions for improving observability are output. If the candidate visual feature set is not empty but the overall score is lower than the preset threshold, a hybrid verification scheme of visual assertion and API verification is generated, and manual review prompts and observability improvement suggestions are output for suggesting the addition of UI features or reviewing the verification results.

5. The method according to claim 1, characterized in that, Based at least on the visual feature assertion description, the test content identified as the UI operation class, and the test content identified as the visual assertion class, a platform-independent structured test decision sequence is generated, including: At least the test content of the UI operation class, the test content of the visual assertion class, and the description of the visual feature assertion are arranged in a structured manner according to the test execution logic to generate a platform-independent structured test decision sequence containing test step identifiers, original test content, intent labels, operation class instructions, assertion class instructions, and execution order information.

6. The method according to claim 1, characterized in that, The structured test decision sequence is converted into executable API call instructions by the target visual automation framework, and then combined to generate an end-to-end executable visual automation test script. The process further includes: The target visual automation framework executes the interactive operations corresponding to the UI operation classes and / or the assertion verification corresponding to the visual feature assertion descriptions in the visual automation test script.

7. The method according to claim 6, characterized in that, The target visual automation framework executes the interaction operations corresponding to the UI operation classes and / or assertion verifications corresponding to the visual feature assertion descriptions in the visual automation test script, including: Obtain a screenshot of the current interface, and generate a correspondence between interactive elements and icons based on the screenshot; Based on the correspondence and the natural language description of the UI operation class and / or visual feature assertion class, determine the target element; and perform corresponding interactive operations and / or assertion verification based on the target element.

8. The method according to claim 1, characterized in that, A UI-Data knowledge graph is constructed using at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relationship extraction. Previously, this also included: Establish L1 coordinate cache, L2 visual label cache, L3 inference result cache, and L4 visual language model response cache; During the execution of the visual automated test script, the corresponding level cache is queried first. If the cache is hit, it is directly reused and the calculation step is skipped. If the cache is not hit, the calculation is performed and the result is written to the cache. Monitor the execution process of the visual automated test script, capture the execution failure information of the automated test script; identify and clear the cache items at all levels associated with the failure information, and adjust at least one of the following according to the failure information: VLM call parameters, element location threshold, assertion wait timeout time or number of retries; Based on the failure information, the feature confidence value of the corresponding mapping relationship in the UI-Data knowledge graph is reduced, and the failure mode is recorded as a negative sample; based on the updated UI-Data knowledge graph and the adjusted parameters, the structured test decision sequence is regenerated and recombined to generate an executable visual automation test script.

9. A hybrid test case to visual automation script conversion system, characterized in that, include: The graph construction module is used to construct a UI-Data knowledge graph through at least one of the following methods: UI structure extraction, API documentation parsing, front-end code static analysis, and large language model relationship extraction. The UI-Data knowledge graph is used to describe the mapping relationship between UI elements and back-end data states. The intent classification module is used to receive mixed-state test cases described in natural language. It adopts a BERT-LLM hybrid two-layer architecture to classify the intent of a single test content in the test case and identify at least one of the three types of intent tags: UI operation, visual assertion, and implicit state. The causal reasoning module is used to perform entity recognition on test content identified as latent state types, extracting the business entities contained therein, the business attributes corresponding to the business entities, and the target state values ​​that need to be verified for the business attributes; using the extracted business entities and the target state values ​​as query conditions, it performs a reverse state propagation path search in the UI-Data knowledge graph to track UI elements that have a direct or indirect relationship with the latent state, and generates a candidate visual feature set based on the searched UI elements; it calculates the score of each visual feature in the candidate visual feature set under multiple evaluation indicators, including visual saliency, feature confidence, detection stability, and historical execution success rate; it calculates the comprehensive score of each visual feature based on the scores of each evaluation indicator, sorts them from high to low comprehensive scores, selects a preset number of visual features with the highest comprehensive scores, and generates corresponding visual feature assertion descriptions based on the selected visual features; The script generation module is used to generate a platform-independent structured test decision sequence based at least on the visual feature assertion description, the content identified as the UI operation class, and the content identified as the visual assertion class; and to convert the structured test decision sequence into API call instructions executable by the target visual automation framework through the target visual automation framework, and combine them to generate an end-to-end executable visual automation test script.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing a computer program to implement the steps of the method as described in any one of claims 1-8.

Images