Web automation testing method and device based on perception and arrangement double-layer AI architecture

Abstract

This invention relates to a web automation testing method based on a two-layer AI architecture of perception and orchestration, comprising: an orchestration layer for receiving test tasks, generating and sending operation instructions through LLM dynamic programming, and invoking the perception layer, and integrating a state monitoring engine and a schema validator; a perception layer, communicatively connected to the orchestration module, for performing multimodal analysis using a large visual language model (VLM) based on the received image data to be analyzed and structured analysis instructions, and returning predefined formatted structured data; wherein, the orchestration module is further configured to generate a new sequence of operation instructions based on the structured data returned by the perception module, through a schema validator, to drive the browser execution agent of the execution layer to complete the test task. This invention can achieve a high degree of intelligence while ensuring the absolute reliability of the test execution process. The architecture further introduces a state transition engine and an output verification mechanism to enhance the robustness and generalization ability of the system.

Description

Technical Field

[0001] This invention relates to the field of computer software testing technology, and in particular to a Web automation testing method and apparatus based on a two-layer AI architecture of perception and orchestration. Background Technology

[0002] Web automation testing is a crucial step in ensuring the quality of modern software. However, existing technologies generally suffer from performance bottlenecks and functional blind spots when dealing with highly dynamic and secure web applications.

[0003] Traditional code-driven solutions (such as those based on Selenium or Playwright) require testers to write a large amount of low-level code, resulting in high development and maintenance costs and an inability to handle complex visual verification tasks. Specifically, these solutions rely on DOM element locators (such as CSS selectors or XPath), which are prone to script failure when faced with asynchronous page loading, dynamic rendering, or UI reconstruction, leading to an exponential increase in maintenance costs. Furthermore, for mainstream image-generated CAPTCHAs (such as calculation or point-and-click questions involving character distortion, overlap, and background interference), traditional OCR solutions suffer from a lack of deep semantic reasoning capabilities, causing recognition accuracy to plummet to 60-70%, making it impossible to establish a reliable automated workflow.

[0004] While existing technologies have attempted to address these pain points by incorporating artificial intelligence (AI), fundamental limitations remain:

[0005] 1. Solutions Based on General AI Agents and Their Unreliability and Uncertainty: Some solutions attempt to use pure end-to-end AI agents to directly manipulate the UI. While these solutions possess a certain level of intelligence, their decision-making process is a "black box." In serious enterprise-level testing scenarios requiring high reliability, the unpredictability, instability (often referred to as the "illusion" problem of AI), and the uncertainty of decision paths are fatal. Specifically, AI agents may produce mutated outputs under the same input, making test results difficult to reproduce and trust, failing to meet the stringent requirements of software quality assurance (such as the stability requirements of regression testing in CI / CD pipelines).

[0006] 2. Preliminary Applications of VLM and its Output Constraints and Engineering Limitations: Some attempts have directly applied large visual language models (VLMs) to CAPTCHA recognition, but these typically remain at the level of simple invocation. Due to the lack of effective constraints and engineering design on the VLM output, the returned results are inconsistent in format and unstable in content, often containing a large amount of irrelevant descriptive text, noise interference, or semantic drift, making them unsuitable for direct and reliable parsing and utilization by automated scripts. Therefore, such simple applications cannot form a stable, closed-loop automated process and lack industrial-grade usability. Furthermore, existing VLM applications do not fully integrate spatial reasoning, semantic association, and multi-step logical links, resulting in insufficient generalization ability when processing interactive CAPTCHAs (such as multi-target localization requiring sorting and clicking).

[0007] 3. Subjectivity and inefficiency of traditional visual regression testing: Existing solutions mainly rely on pixel-level comparison (such as pixel-matching) for UI consistency verification. However, this method cannot understand the semantic meaning of changes, resulting in subjective bias (different testers have a difference of 25-30% in judgment) and low efficiency, which cannot keep up with the pace of rapid iteration in modern front-end development.

[0008] In summary, existing technologies, whether traditional solutions or rudimentary AI application solutions, have failed to provide a comprehensive end-to-end solution that combines high intelligence, high reliability, and high efficiency to systematically address the complex visual interaction challenges in web automation testing. There is an urgent need in this field for a new architecture that can combine A1's multimodal cognitive capabilities with the determinism of engineering practice, while simultaneously supporting dynamic state monitoring, structured output constraints, and cross-model compatibility. Summary of the Invention

[0009] The purpose of this invention is to address at least one of the shortcomings of the prior art by providing a Web automation testing method based on a two-layer AI architecture of perception and orchestration.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] Specifically, a web automation testing method based on a two-layer AI architecture of perception and orchestration is proposed, including the following:

[0012] Obtain high-level test tasks;

[0013] The high-level test task is processed by calling a large language model (LLM) through an orchestration layer to obtain a sequence of subtasks. Operation instructions are generated based on the sequence of subtasks, and then the browser execution agent is controlled to perform web interface operations based on the operation instructions through a standardized instruction protocol.

[0014] When performing the Web interface operation, the interface is monitored by the status monitoring engine. When there is a preset interface that needs visual analysis, the image data of the interface is acquired and preprocessed to obtain the image data to be analyzed.

[0015] The orchestration layer sends the image data to be analyzed and the preset structured analysis instructions to the perception layer.

[0016] The perception layer performs multimodal analysis on the image data to be analyzed using a large visual language model (VLM) based on the structured analysis instructions, generates structured data in a predefined format that includes key information for performing subsequent operations, and sends the structured data to the orchestration layer.

[0017] After receiving the structured data, the orchestration layer generates a new sequence of operation instructions based on the key information, thereby driving the browser execution agent to continue executing the high-level test task until the high-level test task is completed.

[0018] Furthermore, specifically, the preset structured analysis instructions include at least one of role definition instructions, task decomposition instruction set, output format constraint instructions, and interference filtering parameters. The task decomposition instruction set is an ordered array used to guide the chain-of-thought reasoning process of VLM.

[0019] Furthermore, specifically, the interfaces that are pre-defined as requiring visual analysis include interactive CAPTCHAs or UI layout anomaly verification interfaces.

[0020] Furthermore, specifically, the structured data is in JSON format, and the key information includes the spatial location information, content information, operation sequence information, or quantitative evaluation indicators of the target element in the interface.

[0021] Furthermore, specifically, the process by which the orchestration layer generates a new sequence of operation instructions based on the key information after receiving the structured data includes:

[0022] After receiving the structured data, the orchestration layer first performs a schema-based validation step: using a predefined JSON schema to verify the integrity, type consistency, and semantic validity of the structured data. If the validation passes, a transformation step is performed: the structured data is converted from a semantic data representation, i.e., a JSON object, into one or more syntactic operation instruction sequences that can be executed on the browser execution proxy and conform to the standardized instruction protocol. Subsequently, the operation instruction sequence is sent to the browser execution proxy to complete the corresponding operation. If the validation fails, a retry mechanism is triggered. If the retry mechanism succeeds, the process proceeds to the transformation step; if the retry mechanism fails, an alarm is issued.

[0023] Furthermore, specifically, the conversion step is implemented based on a semantic-to-syntactic instruction conversion algorithm, which includes:

[0024] Initialize an empty command queue and a temporary state dictionary execution_state;

[0025] Extract the ordered target sequence target_sequence and the unordered position map hash table position_map from the structured data of the JSON object, while checking the sequence validation field;

[0026] Using a for loop to iterate through target_sequence, one iteration process is as follows:

[0027] i. During iteration, retrieve the element target_char from target_sequence;

[0028] ii. Obtain the logical position (logical_position) in the position_map using target_char as the key;

[0029] iii. Set a conversion function in advance, which takes logical_position as input and generates the final syntactic selector based on the preset CSS selector template;

[0030] iv. Encapsulate the generated syntactic selector into an instruction object conforming to the MCP protocol, push it into the command_queue, and update the execution_state to record the clicked position to avoid duplication;

[0031] After the for loop completes the traversal of target_sequence, the output command_queue is the sequence of syntactic operation instructions.

[0032] Furthermore, specifically, the image data of the interface is acquired and preprocessed to obtain the image data to be analyzed, including:

[0033] The interface is continuously captured according to a preset screenshot frame to obtain a sequence of screenshot images;

[0034] The screenshot image sequence is subjected to image fusion processing to obtain a fused analysis image;

[0035] The fused analysis image is preprocessed to obtain the image data to be analyzed.

[0036] Furthermore, specifically, the process of performing image fusion processing on the screenshot image sequence to obtain a fused analysis image includes,

[0037] If the screenshot image sequence is [img_1, img_2, ..., img_P], and i∈[1, P];

[0038] The initial fused image is obtained by performing a global equal-weighted average on the screenshot image sequence;

[0039] Based on the advantage display index, the screenshot image sequence is traversed to find the advantage display area of ​​img_i;

[0040] The advantageous display areas of all elements in the screenshot image sequence are replaced with the corresponding areas of the initial fused image in ascending order of i, thereby obtaining the fused analysis image;

[0041] The concept of the Advantage Demonstration Index is explained below:

[0042] For img_i, if Let be the pixel value of the pixel in the p-th row and q-th column of the two-dimensional matrix of the image img_i. Advantages Display Index The calculation formula is:

[0043] ;

[0044] in, , as well as They are respectively The average, maximum, and minimum pixel values ​​of the 8 neighboring pixels in the two-dimensional matrix of the image img_i.

[0045] Furthermore, specifically, the screenshot image sequence is traversed based on the advantage display index to identify the advantageous display area of ​​img_i, including,

[0046] When the value of i is 1, that is, for the first element in the screenshot image sequence, traverse all the pixels of img_1, find the two pixels with the largest advantage display index and denot them as feature_1 and feature_2, generate a straight line feature_12 based on feature_1 and feature_2, then find the pixel with the largest advantage display index feature_3 from the pixels of img_1 other than feature_12, and connect feature_1, feature_2 and feature_3 in sequence to obtain the closed area, which is the advantage display area of ​​img_1;

[0047] When the value of i is not 1, first calculate the average pixel value of the pixels in the dominant display area of ​​img_i-1 and denote it as Pj_i-1. Traverse all pixels of img_i and randomly find two pixels with a pixel value higher than Pj_i-1 and denote them as feature_4 and feature_5. Generate a straight line feature_45 based on feature_4 and feature_5. Then, find the pixel with the largest dominant display index feature_6 from the pixels in img_i other than feature_45. Connect feature_4, feature_5 and feature_6 in sequence to obtain the closed area, which is the dominant display area of ​​img_i.

[0048] This invention also proposes a Web automation testing device based on a two-layer AI architecture of perception and orchestration, comprising the following:

[0049] The orchestration layer is configured to receive test tasks, generate and send operation instructions through LLM dynamic programming, call the perception layer, and integrate a state monitoring engine and a schema validator.

[0050] The perception layer, which is communicatively connected to the orchestration module, is configured to perform multimodal analysis using a large visual language model (VLM) based on the received image data to be analyzed and structured analysis instructions, and return structured data in a predefined format.

[0051] The orchestration module is further configured to generate a new sequence of operation instructions based on the structured data returned by the perception module, using a Schema Validator, to drive the browser execution proxy in the execution layer to complete the test task.

[0052] The beneficial effects of this invention are as follows:

[0053] This invention proposes a Web automation testing method and device based on a two-layer AI architecture of perception and orchestration. By decoupling "perception-orchestration", using a deterministic MCP protocol and schema verification mechanism, the risk of uncontrollable behavior of end-to-end AI agents is avoided from the architectural perspective.

[0054] By introducing the "multi-layer structured analysis instruction (Prompt)" designed in this invention, this invention transforms VLM from an unstable "black box" tool into a precise and reliable "visual analysis engine," enabling it to stably output structured data that can be directly used by machines.

[0055] By using image fusion to improve image clarity from the image source as much as possible, weaken the visual impact of fixed minor defects, and eliminate random noise as much as possible, the accuracy of the perceptual layer's multimodal analysis of the image data to be analyzed by the large visual language model (VLM) based on structured analysis instructions can be improved to a certain extent.

[0056] In summary, this invention achieves a high degree of intelligence in Web automated testing while ensuring absolute reliability in the test execution process. Attached Figure Description

[0057] The above and other features of this disclosure will become more apparent from the detailed description of the embodiments illustrated in conjunction with the accompanying drawings. In the accompanying drawings, the same reference numerals denote the same or similar elements. Obviously, the drawings described below are merely some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort. In the drawings:

[0058] Figure 1 The diagram shown is the overall architecture of the Web automation testing method based on a two-layer AI architecture of perception and orchestration according to the present invention.

[0059] Figure 2 The diagram shows a flowchart of the Web automation testing method based on a two-layer AI architecture of perception and orchestration according to the present invention.

[0060] Figure 3 The diagram shown is a flowchart of the conversion process from JSON to MCP instructions in this invention.

[0061] Figure 4 The image shown is a schematic diagram of an example of the character verification code involved in this invention.

[0062] Figure 5 The diagram shown is an example of an interface schematic of the dynamic mathematical calculation involved in this invention.

[0063] Figure 6 The image shown is a schematic diagram of an example of Chinese sequential selection involved in this invention.

[0064] Figure 7 The image shown is a schematic diagram of an example of a damaged page involved in this invention.

[0065] Figure 8 The diagram shown is an example of an interface diagram illustrating a color error involved in this invention. Detailed Implementation

[0066] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The same reference numerals used throughout the accompanying drawings indicate the same or similar parts.

[0067] Example 1, referring to Figure 1 , Figure 2 as well as Figure 3 This invention proposes a web automation testing method based on a two-layer AI architecture of perception and orchestration, including the following:

[0068] Obtain high-level test tasks;

[0069] The high-level test task is processed by calling a large language model (LLM) through an orchestration layer to obtain a sequence of subtasks. Operation instructions are generated based on the sequence of subtasks, and then the browser execution agent is controlled to perform web interface operations based on the operation instructions through a standardized instruction protocol.

[0070] When performing the aforementioned web interface operations, the interface is monitored through a state monitoring engine. When there are preset interfaces that require visual analysis (such as...) Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 When (as shown), the image data of the interface is acquired and preprocessed to obtain the image data to be analyzed;

[0071] The orchestration layer sends the image data to be analyzed and the preset structured analysis instructions to the perception layer.

[0072] The perception layer performs multimodal analysis on the image data to be analyzed using a large visual language model (VLM) based on the structured analysis instructions, generates structured data in a predefined format that includes key information for performing subsequent operations, and sends the structured data to the orchestration layer.

[0073] After receiving the structured data, the orchestration layer generates a new sequence of operation instructions based on the key information, thereby driving the browser execution agent to continue executing the high-level test task until the high-level test task is completed.

[0074] In this embodiment 1, the perception layer, responsible for semantic extraction and reasoning of multimodal visual information, is functionally decoupled from the orchestration layer, which is responsible for dynamic task planning, state monitoring, and execution orchestration. Communication and coordination are achieved through a standardized, deterministic instruction protocol (such as the Model Control Protocol, MCP), thereby ensuring absolute reliability of the test execution process while achieving high intelligence. This architecture further introduces a state transition engine and output verification mechanism to enhance the system's robustness and generalization capabilities.

[0075] Compared with existing technologies, this invention, through its unique two-layer decoupling architecture, state monitoring engine, and multi-dimensional constraint mechanism, brings the following experimentally verified significant technological advancements:

[0076] 1. This invention addresses the reliability and non-deterministic challenges of AI testing: By employing a decoupled "perception-orchestration" approach, a deterministic MCP protocol, and a schema verification mechanism, it architecturally mitigates the risk of uncontrollable behavior in end-to-end AI agents. Experiments demonstrate that this system achieves a high success rate (zero failures in 200 tests) across browsers (Chromium, Firefox, Web Kit) when handling two industry-recognized types of complex CAPTCHAs (dynamic mathematical calculations and Chinese character selection).

[0077] 2. Achieved accurate multimodal understanding and generalization of complex visual scenes: By introducing a carefully designed "multi-layer structured analysis instruction (Prompt)," this invention transforms VLM from an unstable "black box" tool into a precise and reliable "visual analysis engine," enabling it to stably output structured data that can be directly used by machines, while supporting spatial reasoning and semantic association. This is something that existing simple AI application solutions cannot achieve. Furthermore, the system surpasses pixel contrast in visual regression tests, achieving quantitative evaluation of UI layout, color contrast, and readability (such as calculating overlapping pixel area and WCAG contrast ratio).

[0078] 3. Significantly Improved Efficiency, Scope, and Scalability of Automated Testing: This invention significantly improves the development efficiency of automated testing. In a typical comparative test case involving the creation of complex business processes, the traditional manual script writing process takes approximately 20 minutes, while using this invention, driven by natural language intent, the same task can be planned and executed in only about 3.5 minutes, improving single-point task development efficiency by approximately 82.5%. Furthermore, this invention, for the first time, extends the capabilities of automated testing from traditional functional verification to the objective and automated evaluation of UI / UX quality, greatly broadening the application boundaries of automated testing. It also supports open-source model compatibility (such as Kimi-K2) and achieves "zero-code" natural language-driven processing in end-to-end business processes, lowering the technical threshold.

[0079] The specific operating steps of this invention are as follows:

[0080] 1. Task Reception and Dynamic Planning (Orchestration Layer): The orchestration layer receives a high-level test task (which can be a natural language intent) and, using its built-in Large Language Model (LLM), autonomously plans and decomposes the task into a series of ordered, atomized subtask sequences through a chain-of-thought mechanism. This planning process includes internal state modeling (such as to-do list generation) and conditional branch prediction to adapt to dynamic page changes.

[0081] 2. Instruction Generation and Execution (Orchestration Layer -> Execution Agent): For each subtask, the orchestration layer transforms it into a deterministic operation instruction conforming to a standardized instruction protocol (such as MCP) and sends it to the browser execution agent to complete the actual interaction with the web browser. This protocol defines a toolset (such as browser_navigate, browser_click, browser_type) to ensure the atomicity and security of instructions. Simultaneously, a feedback loop is introduced: the execution agent returns the operation result (such as page state changes), allowing the orchestration layer to adjust subsequent planning in real time.

[0082] 3. Visual Challenge Recognition and Perception Layer Invocation (Collaboration Core): During execution, the state monitoring engine within the orchestration layer (based on DOM polling or Mutation Observer) monitors the page state in real time. When it identifies a dynamically generated, image-recognition-based CAPTCHA on the current interface, or a UI area suspected of having layout anomalies, the visual analysis sub-process is triggered. The orchestration layer takes a screenshot of the interface or element, obtains image data, and performs preprocessing (such as Base64 encoding and noise filtering) to optimize subsequent analysis efficiency.

[0083] 4. Multi-dimensional Analysis and Reasoning in the Perception Layer: The orchestration layer constructs a request from the image data and a pre-defined structured analysis instruction (Prompt) corresponding to the task type, and sends it to the perception layer. The core of the perception layer is a large visual language model (VLM), which performs multimodal analysis on the image data according to the structured instruction, including pixel-level recognition, semantic reasoning, and spatial relationship mapping, and returns the analysis results in a predefined, deterministic structured data format (such as JSON).

[0084] The types of image recognition-based CAPTCHAs include, but are not limited to: computational CAPTCHAs that require mathematical operations, or point-selection CAPTCHAs that require sequential spatial positioning.

[0085] The structured analysis instruction is a multi-level nested data structure used to constrain and guide VLM output, which contains one or more of the following fields:

[0086] Role definition field: Used to set the analysis mode of VLM (e.g., set as "mathematical expert" or "visual spatial positioning expert") to activate specific knowledge subgraphs within the model.

[0087] Task decomposition instruction set: an ordered array containing multiple sub-instructions used to guide the logical reasoning process of VLM (such as "1. Identify target sequence; 2. Map grid position; 3. Filter interference"), and improve reasoning accuracy by utilizing the Chain-of-Thought principle.

[0088] Output format constraint fields define the schema of the returned structured data, including field types, required fields, and validation rules, enforcing consistent output format to ensure machine parsing. Simultaneously, a fault tolerance threshold is introduced, allowing limited noise while mandating the integrity of core fields.

[0089] Interference filtering parameters: Optional field used to specify the processing strategy for background noise, distortion, or multilingual elements, improving robustness in complex scenarios.

[0090] 5. Result Transformation, Validation, and Execution Closed Loop: After receiving the structured data, the orchestration layer first performs a schema-based validation step: using a predefined JSON schema to verify its integrity, type consistency, and semantic validity (e.g., whether the result is within the expected range). If validation passes, a crucial transformation step is performed: converting it from a semantic data representation (JSON object) into one or more syntactic command sequences that can be executed on the browser execution agent and conform to the aforementioned standardized command protocol. This transformation involves hash mapping, loop iteration, and selector generation algorithms. Subsequently, the newly generated command sequence is sent to the execution agent to complete the precise operation, and the system state is updated based on feedback, forming a complete automated closed loop of "perception-decision-execution-validation." If validation fails, a retry mechanism is triggered (e.g., adjusting the Prompt parameter or switching to a backup VLM model).

[0091] As a preferred embodiment of the present invention, specifically, the preset structured analysis instructions include at least one of role definition instructions, task decomposition instruction set, output format constraint instructions, and interference filtering parameters. The task decomposition instruction set is an ordered array used to guide the chain-of-thought reasoning process of VLM.

[0092] As a preferred embodiment of the present invention, specifically, the preset interfaces requiring visual analysis include interactive verification codes or UI layout anomaly verification interfaces.

[0093] In a preferred embodiment of the present invention, the structured data is in JSON format, and the key information includes the spatial location information, content information, operation sequence information, or quantitative evaluation indicators of the target element in the interface.

[0094] In a preferred embodiment of the present invention, specifically, the process by which the orchestration layer generates a new sequence of operation instructions based on the key information after receiving the structured data includes:

[0095] After receiving the structured data, the orchestration layer first performs a schema-based validation step: using a predefined JSON schema to verify the integrity, type consistency, and semantic validity of the structured data. If the validation passes, a transformation step is performed: the structured data is converted from a semantic data representation, i.e., a JSON object, into one or more syntactic operation instruction sequences that can be executed on the browser execution proxy and conform to the standardized instruction protocol. Subsequently, the operation instruction sequence is sent to the browser execution proxy to complete the corresponding operation. If the validation fails, a retry mechanism is triggered. If the retry mechanism succeeds, the process proceeds to the transformation step; if the retry mechanism fails, an alarm is issued.

[0096] In a preferred embodiment of the present invention, the conversion step is specifically implemented based on a semantic-to-syntactic instruction conversion algorithm, which includes:

[0097] Initialize an empty command queue and a temporary state dictionary execution_state;

[0098] Extract the ordered target sequence target_sequence and the unordered position map hash table position_map from the structured data of the JSON object, while checking the sequence validation field;

[0099] Using a for loop to iterate through target_sequence, one iteration process is as follows:

[0100] i. During iteration, retrieve the element target_char from target_sequence;

[0101] ii. Obtain the logical position (logical_position) in the position_map using target_char as the key;

[0102] iii. Set a conversion function in advance, which takes logical_position as input and generates the final syntactic selector based on the preset CSS selector template;

[0103] iv. Encapsulate the generated syntactic selector into an instruction object conforming to the MCP protocol, push it into the command_queue, and update the execution_state to record the clicked position to avoid duplication;

[0104] After the for loop completes the traversal of target_sequence, the output command_queue is the sequence of syntactic operation instructions.

[0105] In a preferred embodiment of the present invention, specifically, image data of the interface is acquired and preprocessed to obtain image data to be analyzed, including,

[0106] The interface is continuously captured according to a preset screenshot frame to obtain a sequence of screenshot images;

[0107] The screenshot image sequence is subjected to image fusion processing to obtain a fused analysis image;

[0108] The fused analysis image is preprocessed to obtain the image data to be analyzed.

[0109] In this preferred embodiment, by using image fusion to improve image clarity from the image source as much as possible, weaken the visual impact of fixed minor defects, and eliminate random noise as much as possible, the accuracy of the results of the perception layer performing multimodal analysis on the image data to be analyzed based on the large visual language model (VLM) according to structured analysis instructions can be improved to a certain extent.

[0110] In a preferred embodiment of the present invention, specifically, the process of performing image fusion processing on the screenshot image sequence to obtain a fused analysis image includes:

[0111] If the screenshot image sequence is [img_1, img_2, ..., img_P], and i∈[1, P];

[0112] The initial fused image is obtained by performing a global equal-weighted average on the screenshot image sequence;

[0113] Based on the advantage display index, the screenshot image sequence is traversed to find the advantage display area of ​​img_i;

[0114] The advantageous display areas of all elements in the screenshot image sequence are replaced with the corresponding areas of the initial fused image in ascending order of i, thereby obtaining the fused analysis image;

[0115] The concept of the Advantage Demonstration Index is explained below:

[0116] For img_i, if Let be the pixel value of the pixel in the p-th row and q-th column of the two-dimensional matrix of the image img_i. Advantages Display Index The calculation formula is:

[0117] ;

[0118] in, , as well as They are respectively The average, maximum, and minimum pixel values ​​of the 8 neighboring pixels in the two-dimensional matrix of the image img_i.

[0119] In this preferred embodiment, considering that existing image fusion methods often obtain a fused image by performing a global weighted average of a group of images (i.e., averaging the pixel values ​​at the same pixel positions in the group of images), although this method can balance some random noise, it is relatively poor in cases where the sharpness of different frames fluctuates. It may weaken pixels with high sharpness in some frames due to the averaging of pixel values. The image fusion method proposed in this preferred embodiment can solve this problem well, thereby significantly improving the image quality of the input VLM and ensuring the accuracy of the results of multimodal analysis of the image data to be analyzed by the subsequent perception layer through the large visual language model VLM according to the structured analysis instructions.

[0120] In a preferred embodiment of the present invention, specifically, the screenshot image sequence is traversed based on the advantage display index to identify the advantageous display area of ​​img_i, including...

[0121] When the value of i is 1, that is, for the first element in the screenshot image sequence, traverse all the pixels of img_1, find the two pixels with the largest advantage display index and denot them as feature_1 and feature_2, generate a straight line feature_12 based on feature_1 and feature_2, then find the pixel with the largest advantage display index feature_3 from the pixels of img_1 other than feature_12, and connect feature_1, feature_2 and feature_3 in sequence to obtain the closed area, which is the advantage display area of ​​img_1;

[0122] When the value of i is not 1, first calculate the average pixel value of the pixels in the dominant display area of ​​img_i-1 and denote it as Pj_i-1. Traverse all pixels of img_i and randomly find two pixels with a pixel value higher than Pj_i-1 and denote them as feature_4 and feature_5. Generate a straight line feature_45 based on feature_4 and feature_5. Then, find the pixel with the largest dominant display index feature_6 from the pixels in img_i other than feature_45. Connect feature_4, feature_5 and feature_6 in sequence to obtain the closed area, which is the dominant display area of ​​img_i.

[0123] In this preferred embodiment, it can be clearly seen that the algorithm starts iterating from the first image, and each subsequent iteration is based on the advantageous display area of ​​the first image. This ensures that the advantageous display areas selected in subsequent images are better than those in the previous image. As a result, the region replacement of the initial fused image obtained by globally weighted averaging the screenshot image sequence always maintains the advantage iteration. Moreover, this method has a small computational load, high efficiency, and the fused image obtained has a high degree of detail in the content.

[0124] The following are several embodiments of the Web automation testing method based on a two-layer AI architecture of perception and orchestration proposed in this invention:

[0125] Example 1: Automated processing of dynamic mathematical calculation CAPTCHAs (including interference filtering and inference chain)

[0126] This embodiment demonstrates how the system handles a dynamic mathematical calculation CAPTCHA with visual interference (such as distorted lines and background noise), emphasizing the integration of multi-step inference links.

[0127] 1. State Monitoring and Decision Triggering (Orchestration Layer): In the system of this invention, a state monitoring engine runs internally within the orchestration layer module. This engine combines DOM Mutation Observer and a timed polling mechanism to monitor changes in page elements in real time. When the rule matching module of this engine detects that preset triggering conditions are met (e.g., the return value of document.querySelector('#captcha-image') is not empty and the element is visible, while checking whether its attributes such as src are dynamically updated), the system state transitions from 'normal execution' to 'visual analysis standby'. At this time, the orchestration layer pauses the execution of the main task queue and starts the visual analysis sub-process, while recording the current context state to support subsequent recovery.

[0128] 2. Image Data Acquisition and Preprocessing (Orchestration Layer): In the visual analysis sub-flow, the orchestration layer first sends an instruction to the execution agent to precisely capture the viewport region of the challenge element (#captcha-image), rather than the entire page, to reduce interference from irrelevant information. The acquired raw image data (such as PNG format) undergoes standardized preprocessing, such as uniformly converting it to a Base64 encoded string, and applies optional noise filtering (such as grayscale conversion or edge enhancement) to optimize the input quality of the VLM.

[0129] 3. Dynamic Construction of Structured Analysis Instructions (Orchestration Layer): Instead of sending a static text string, the orchestration layer dynamically constructs a structured request payload. At its core is a JSON object, whose 'Prompt' field is itself a complex data structure containing multi-dimensional constraints. This structure aims to minimize the stochasticity of the VLM and guide it into a deterministic analysis path. This data structure includes:

[0130] The .role_definition field: Its value (such as 'mathematical expert') is used to boost the weights of neurons related to mathematics and logical reasoning in the attention mechanism of VLM.

[0131] The `task_decomposition` field is an array of instructions that explicitly breaks down a macro-level task into a sequence of micro-steps that the VLM must follow (e.g., "1. Accurately identify mathematical characters of various fonts, sizes, and colors; 2. Understand the complete semantics of mathematical expressions, rather than simple character recognition; 3. Automatically filter background noise, interfering lines, and other irrelevant information; 4. Identify mathematical expressions in images and calculate accurate results"). This leverages the "chain-of-thought" principle in AI, forcing the VLM to simulate a logically coherent thought process, thereby improving the accuracy of the final result.

[0132] The `output_schema_constraint` field: This is not a simple formatting requirement, but a strict JSON schema definition, including type constraints (such as "result" must be a number) and required fields. Upon receiving the VLM's response, the orchestration layer first uses this schema to validate the returned JSON. Any response that does not conform to this schema (such as missing fields, incorrect types, or out-of-range values) will be considered invalid and trigger a retry or error handling mechanism (such as adding interference filtering parameters). This design is crucial for ensuring the robustness of the entire system.

[0133] The .interference_filter_params field is an optional parameter used to specify a particular type of interference (such as "Number of Twisted Lines: High"), which guides the VLM to prioritize filtering relevant noise.

[0134] An example of the constructed request payload is as follows:

[0135] {

[0136] "image_data":"iVBORw0KGgoAAAANSUhEUgAA...", / / Base64 encoded image

[0137] "prompt":{

[0138] "role_definition":"You are a professional mathematical expression recognition expert..."

[0139] "task_decomposition":[

[0140] "Accurately identifies mathematical characters of various fonts, sizes, and colors."

[0141] "Understanding the full semantics of mathematical expressions, rather than simple character recognition."

[0142] Automatically filters out background noise, distracting lines, and other irrelevant information.

[0143] "Identify mathematical expressions in images and calculate accurate results."

[0144] ],

[0145] "output_schema_constraint":{

[0146] "type":"object",

[0147] "properties":{

[0148] "expression":{"type":"string"},

[0149] "calculation":{"type":"string"},

[0150] "result":{"type":"number"}

[0151] },

[0152] "required":["expression","result"],

[0153] "additional Properties":false

[0154] },

[0155] "interference_filter_params":{

[0156] "noise_level":"high",

[0157] "distortion_type":"lines_and_blur"

[0158] }

[0159] }

[0160] }

[0161] 4. Parsing, Transformation, and Execution (Orchestration Layer -> Execution Agent): The perception module's VLM (such as Qwen-VL) analyzes the data and returns a JSON result, for example: {"expression":"45-12=?","calculation":"45 minus 12 equals 33","result":33}. Upon receiving this, the orchestration module first validates the result against a predefined schema (including range checks, such as whether the result is an integer). If validation passes, it parses the JSON and extracts the value 33 from the result field. Finally, it converts this semantic result (the answer "33") into a deterministic syntactic instruction (entering the string "33" into the input box with ID captcha-input) and encapsulates it as an MCP instruction before sending it to the execution agent. Then, the main task queue is restored, and the "Login" button click operation continues to complete the verification. If multiple calculations are involved, the system can recursively call the VLM to handle complex expressions.

[0162] Example 2: Automated processing of Chinese interactive CAPTCHA (including spatial mapping and sequential execution):

[0163] This embodiment demonstrates how the system handles a complex interactive CAPTCHA that requires clicking on specified Chinese characters in sequence, emphasizing the integration of spatial mapping algorithms and sequential execution chains.

[0164] 1. Status Monitoring and Decision Triggering (Orchestration Layer): Similar to Example 1, the status monitoring engine in the orchestration layer identifies Chinese point-and-click verification codes through rule matching (e.g., detecting the .captcha-container element and its sub-mesh structure). The system status then transitions to 'Visual Analysis Standby', and the main task queue is paused. Simultaneously, the engine records the mesh size (e.g., 4x4) to support subsequent transformations.

[0165] 2. Image Data Acquisition and Preprocessing (Orchestration Layer): The orchestration layer instruction execution agent captures the PNG image of the .captcha-container element and converts it into a Base64 encoded string. To optimize Chinese character recognition, preprocessing includes character sharpening or multilingual font enhancement.

[0166] 3. Dynamic construction of structured analysis instructions (orchestration layer): For this type of spatial localization task, the orchestration layer constructs structured requests that focus on visual localization and logical reasoning.

[0167] Its prompt field's data structure includes:

[0168] The role_definition field is set to "Expert in visual spatial positioning proficient in Chinese" to activate VLM's capabilities in Chinese character recognition and spatial relationship judgment.

[0169] The task_decomposition field: The instruction array explicitly requires VLM to perform three steps in sequence: "identify the target character sequence", "identify the 16 Chinese characters in all grids", and "accurately match the grid position of each target character from 1 to 16, taking into account sorting constraints", forming a rigorous "thinking chain".

[0170] The `output_schema_constraint` field strictly defines that the output JSON must contain `target Chars` (an ordered array), `grid Mapping` (a key-value pair object), and `sequence Validation` (a boolean field for validating the sorting), ensuring the integrity and consistency of the data required for subsequent algorithm processing.

[0171] The `interference_filter_params` field specifies Chinese-specific interference, such as "adhesive characters" or "font distortion".

[0172] 4. Semantic-to-syntactic instruction conversion algorithm (Orchestration layer): After receiving and validating JSON data from the perception layer (e.g., {"target Chars":["water","cloud"],"grid Mapping":{"water":5,"":14},"sequence Validation":true}), the orchestration layer initiates a core semantic-to-syntactic conversion algorithm. The goal of this algorithm is to convert a cognitive result describing "what" into a deterministic sequence of machine instructions describing "how," while ensuring sequentiality and atomicity.

[0173] The algorithm execution flow is as follows:

[0174] a. Initialize an empty command queue (command_queue) and a temporary state dictionary (execution_state) (for tracking click order).

[0175] b. Parse semantic data: Extract the ordered target sequence `target_sequence=["water","cloud"]` and the unordered position mapping hash table `position_map={"water":5,"cloud":14}` from the JSON object. Simultaneously, check the `sequenceValidation` field; if it is false, trigger a retry.

[0176] c. Iteration and transformation: The system iterates through the target_sequence array using a for loop.

[0177] i. In the first iteration, set target_char to "water".

[0178] ii. Perform a critical hash lookup: In the position_map, use target_char as the key to obtain its logical position (logical_position=5). If the lookup fails, log the error and roll back the state.

[0179] iii. Apply predefined syntactic transformation rules: The system has a built-in transformation function that takes the logical position and generates the final syntactic selector based on the preset CSS selector template (e.g., .char-button:nth-child({position})). Here, the input is 5, and the output is ".char-button:nth-child(5)". At the same time, browser compatibility (e.g., WebKit's nth-child variant) is considered.

[0180] iv. Encapsulate as a standardized command: Encapsulate the generated selector into a command object conforming to the MCP protocol, such as {tool:'browser_click',selector:'.char-button:nth-child(5)',options:{delay:200}} (add a delay to simulate human behavior), and push it into the command_queue. Update execution_state to record the clicked position to avoid duplication.

[0181] d. The loop continues until the target_sequence has been traversed. The command_queue will then contain an ordered, deterministic list of commands.

[0182] 5. Serialization Execution and Feedback Loop (Orchestration Layer -> Execution Agent): Finally, the orchestration layer sends the instructions in the command_queue one by one to the execution agent according to the first-in, first-out (FIFO) principle, ensuring the sequentiality and atomicity of the operations. After each instruction is executed, the agent returns feedback (such as a successful click or page change) for the orchestration layer to verify. If the feedback is abnormal, a rollback or alternative path is triggered, thereby completing complex interactive tasks.

[0183] Example 3: End-to-end business process automation driven by natural language (including model compatibility and performance optimization)

[0184] This embodiment emphasizes cross-model compatibility and performance optimization.

[0185] 1. Receiving a natural language task: The tester inputs a natural language instruction into the orchestration module:

[0186] “Use the Play Wright MCP tool to navigate to the specified website, log in with admin / password, then create a new employee named ‘Zhang San’ with a salary of 80,000, and finally take a screenshot for verification.”

[0187] 2. LLM Autonomous Planning and Decomposition: Upon receiving the instruction, the LLM within the orchestration module (such as Claude4Sonnet or the compatible Kimi-K2) autonomously plans and decomposes it into an internal, ordered to-do list, for example:

[0188] [ ] Navigate to the specified website

[0189] [Click the login link (if a verification code exists, trigger the perception layer)]

[0190] Enter 'admin' in the username field. []

[0191] Enter 'password' in the password field. []

[0192] [Click the login button]

[0193] [Click the link to create a new employee]

[0194] Enter 'Zhang San' in the name field. []

[0195] Enter '80000' in the salary field.

[0196] [Click the Create button]

[0197] Save the screenshot as 'employee-creation-success.png' and verify the consistency of the content.

[0198] The decomposition process supports model switching: if the open-source model Kimi-K2 is used, the system automatically adjusts the Prompt to optimize its Agentic task capabilities.

[0199] 3. MCP Instruction Execution and Optimization: Following the list above, the LLM sequentially translates each subtask into a corresponding MCP tool call and sends it to the execution agent. For example, the "Navigation" task is translated into the instruction `mcp__playwright__browser_navigate(url="local host")`, and the "Input" task is translated into the instruction `mcp__playwright__browser_type(selector="#Name",text="Zhang San")`. Performance optimizations are introduced, such as batch instruction merging (combining continuous input into a single call) and a timeout retry mechanism.

[0200] 4. Completion and Verification: After the agent completes all instructions, the entire business process is automated. The final screenshot step provides visual evidence of the test's success. The entire process requires no manual coding and is completely driven by AI, supporting multiple rounds of testing (such as 2*5 rounds) and success rate statistics.

[0201] Example 4: Automated Intelligent Assessment of UI / UX Defects (including quantitative indicators and extended assessment)

[0202] This embodiment demonstrates a specific application of the system's scalability, emphasizing the integration of quantitative indicators.

[0203] 1. Task Definition: The test task is to "evaluate the UI layout quality of the login page, including layout, color and readability".

[0204] 2. Invoking the Perception Layer: The orchestration layer takes a screenshot of the entire login page and invokes the perception layer using the following structured analysis instruction (Prompt). This instruction also follows the structured request payload design described in Examples 1 and 2, and its prompt field contains explicit role_definition (UI / UX expert), task_decomposition (detecting overlap, truncation, and contrast), and output_schema_constraint (strict JSON schema, including quantitative fields such as overlap_pixels).

[0205] You are a W3C-compliant UI / UX expert. Please perform a layout and accessibility analysis on the provided page screenshots.

[0206] 1. Detect the overlap of all interface elements and estimate the overlap area (in pixels) using approximate bounding boxes.

[0207] 2. Identify all text elements that are truncated or overflow the container, and estimate the overflow percentage.

[0208] 3. Check the color contrast of all text with its background, mark items that are lower than the WCAGAA standard (4.5:1), and calculate the actual contrast ratio.

[0209] 4. Evaluate the overall color harmony and readability, and give a score of 1-10.

[0210] Please return the analysis results in the following JSON format:

[0211] "issues":[

[0212] {

[0213] "type":"OVERLAP",

[0214] "elements":["#username_input","#login_button"],

[0215] "overlap_pixels":150,

[0216] "description":"User name input field over laps with the loginbutton."

[0217] },

[0218] {

[0219] "type":"CONTRAST",

[0220] "element":"span.forgot-password",

[0221] "contrast_ratio":2.1,

[0222] "description":"Forgot password link has in sufficient contrast."

[0223] }

[0224] ],

[0225] "overall_score": 6.5

[0226] }

[0227] 1. Report Generation and Extension: After VLM analysis by the perception layer, a JSON report containing specific defects, quantitative metrics, and overall scores is returned. The orchestration layer can integrate this report into the test results and extend it to multi-dimensional assessments (such as information architecture clarity), providing objective, quantifiable, and automated evaluations of UI quality, rather than just subjective judgments.

[0228] The technical terminology used in this invention is further explained below:

[0229] AI Agent

[0230] An autonomous entity based on artificial intelligence technology, a software system that can perceive the environment, make decisions and perform operations.

[0231] Base64 encoding

[0232] An encoding method that uses 64 characters to represent arbitrary binary data, often used to transmit image data in text protocols.

[0233] Chain of Thought

[0234] An AI reasoning technique that improves reasoning accuracy by breaking down complex problems into a series of logical steps and guiding the model to think in a structured way.

[0235] CSS Selectors

[0236] A pattern used to select a specific element in an HTML document, such as #username representing the element with the ID username.

[0237] DOM (Document Object Model)

[0238] The Document Object Model (DOM) is a programming interface for HTML and XML documents, allowing programs to dynamically access and update document content.

[0239] Hallucination

[0240] The phenomenon that AI models generate inaccurate, non-existent, or inconsistent information is a common problem in large language models.

[0241] JSON Schema

[0242] A specification for validating the structure and content of JSON data, defining the data type, format, and constraints.

[0243] LLM (Large Language Model)

[0244] Large-scale language models, natural language processing models trained using deep learning, such as GPT and Claude.

[0245] MCP (Model Control Protocol)

[0246] Model control protocol, a standardized instruction format used for communication between AI models and external tools.

[0247] Mutation Observer

[0248] Browser APIs are used to monitor changes in the DOM tree and trigger callback functions when page elements are added, deleted, or modified.

[0249] OCR (Optical Character Recognition)

[0250] Optical character recognition technology converts text in an image into an editable text format.

[0251] Playwright

[0252] Microsoft's automated testing framework supports cross-browser web application testing.

[0253] Selenium

[0254] An open-source web application automation testing tool that controls the browser via the WebDriver protocol.

[0255] Schema Validation

[0256] The process of verifying whether the data format, type, and content meet the requirements according to the predefined data structure specifications.

[0257] VLM (Vision Language Model)

[0258] Visual language models are multimodal AI models capable of processing both image and text information simultaneously.

[0259] WCAG (Web Content Accessibility Guidelines)

[0260] The Web Content Accessibility Guidelines define international standards for web page accessibility, including requirements such as color contrast.

[0261] Web Kit

[0262] An open-source browser engine for the Safari browser that supports rendering HTML, CSS, and JavaScript.

[0263] X Path

[0264] XML Path Language is a query language used to select nodes in XML / HTML documents.

[0265] Orchestration Layer

[0266] The AI ​​architecture layer in this invention, which is responsible for task planning, status monitoring, and execution orchestration, includes an LLM and a status monitoring engine.

[0267] Perception Layer

[0268] The AI ​​architecture layer responsible for visual information processing and semantic extraction in this invention is based on the VLM model.

[0269] Structured Analysis Prompt

[0270] The core innovation of this invention is a multi-layered nested data structure used to constrain and guide VLM output, which includes fields such as role definition, task decomposition, and format constraints.

[0271] Semantic to Syntactic Translation

[0272] The key algorithm of this invention is the process of converting semantic JSON data output by VLM into an executable MCP instruction sequence.

[0273] State Monitoring Engine

[0274] The core components of this invention for real-time monitoring of page state changes and identification of visual challenges are implemented based on DOM polling and Mutation Observer.

[0275] Visual Challenge

[0276] Interface elements that require image recognition and semantic understanding to resolve, such as CAPTCHAs and UI layout anomalies.

[0277] This invention also proposes a Web automation testing device based on a two-layer AI architecture of perception and orchestration, comprising the following:

[0278] The orchestration layer is configured to receive test tasks, generate and send operation instructions through LLM dynamic programming, call the perception layer, and integrate a state monitoring engine and a schema validator.

[0279] The perception layer, which is communicatively connected to the orchestration module, is configured to perform multimodal analysis using a large visual language model (VLM) based on the received image data to be analyzed and structured analysis instructions, and return structured data in a predefined format.

[0280] The orchestration module is further configured to generate a new sequence of operation instructions based on the structured data returned by the perception module, using a Schema Validator, to drive the browser execution proxy in the execution layer to complete the test task.

[0281] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment, depending on actual needs.

[0282] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0283] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or system capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0284] Although the description of the invention has been quite detailed and particularly of several described embodiments, it is not intended to limit it to any of these details or embodiments or any particular embodiment, but should be considered as providing a broad possible interpretation of the claims by referring to the appended claims and taking into account the prior art, thereby effectively covering the intended scope of the invention. Furthermore, the invention has been described above with respect to embodiments foreseeable by the inventors in order to provide a useful description, and non-substantial modifications to the invention that have not yet been foreseen may still represent equivalent modifications.

[0285] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any embodiment that achieves the technical effects of the present invention using the same means should fall within the protection scope of the present invention. Within the protection scope of the present invention, various modifications and variations can be made to the technical solutions and / or implementation methods.

Claims

1. A Web automation testing method based on a two-layer AI architecture of perception and orchestration, characterized in that, Including the following: Obtain high-level test tasks; The high-level test task is processed by calling a large language model (LLM) through an orchestration layer to obtain a sequence of subtasks. Operation instructions are generated based on the sequence of subtasks, and then the browser execution agent is controlled to perform web interface operations based on the operation instructions through a standardized instruction protocol. When performing the Web interface operation, the interface is monitored by the status monitoring engine. When there is a preset interface that needs visual analysis, the image data of the interface is acquired and preprocessed to obtain the image data to be analyzed. The orchestration layer sends the image data to be analyzed and the preset structured analysis instructions to the perception layer. The perception layer performs multimodal analysis on the image data to be analyzed using a large visual language model (VLM) based on the structured analysis instructions, generates structured data in a predefined format that includes key information for performing subsequent operations, and sends the structured data to the orchestration layer. After receiving the structured data, the orchestration layer generates a new sequence of operation instructions based on the key information, thereby driving the browser execution proxy to continue executing the high-level test task until the high-level test task is completed. Specifically, the process by which the orchestration layer generates a new sequence of operation instructions based on the key information after receiving the structured data includes: After receiving the structured data, the orchestration layer first performs a schema-based validation step: using a predefined JSONSchema to verify the integrity, type consistency, and semantic validity of the structured data. After the validation passes, a transformation step is performed: the structured data is converted from a semantic data representation, i.e., a JSON object, into one or more syntactic operation instruction sequences that can be executed on the browser execution agent and conform to the standardized instruction protocol. Subsequently, the operation instruction sequence is sent to the browser execution agent to complete the corresponding operation. If the verification fails, a retry mechanism is triggered. If the retry mechanism succeeds, the process proceeds to the conversion step. If the retry mechanism fails, an alarm is triggered. Specifically, the conversion step is implemented based on a semantic-to-syntactic instruction conversion algorithm, which includes: Initialize an empty command queue and a temporary state dictionary execution_state; Extract the ordered target sequence target_sequence and the unordered position mapping hash table position_map from the structured data of the JSON object, while checking the sequenceValidation field; Using a for loop to iterate through target_sequence, one iteration process is as follows: i. During iteration, retrieve the element target_char from target_sequence; ii. Obtain the logical position (logical_position) in the position_map using target_char as the key; iii. Pre-set a transformation function that takes logical_position as input and generates the final syntactic selector based on a pre-defined CSS selector template; iv. Encapsulate the generated syntactic selector into an instruction object conforming to the MCP protocol, push it into the command_queue, and update the execution_state to record the clicked position to avoid duplication; After the for loop completes the traversal of target_sequence, the output command_queue is the sequence of syntactic operation instructions.

2. The Web automation testing method based on a two-layer AI architecture of perception and orchestration as described in claim 1, characterized in that, Specifically, the preset structured analysis instructions include at least one of the following: role definition instructions, task decomposition instruction set, output format constraint instructions, and interference filtering parameters. The task decomposition instruction set is an ordered array used to guide the chain-of-thought reasoning process of VLM.

3. The Web automation testing method based on a two-layer AI architecture of perception and orchestration as described in claim 1, characterized in that, Specifically, the interfaces that require visual analysis include interactive CAPTCHAs or UI layout anomaly verification interfaces.

4. The Web automation testing method based on a two-layer AI architecture of perception and orchestration as described in claim 1, characterized in that, Specifically, the structured data is in JSON format, and the key information includes the spatial location information, content information, operation sequence information, or quantitative evaluation indicators of the target element in the interface.

5. The Web automation testing method based on a two-layer AI architecture of perception and orchestration as described in claim 1, characterized in that, Specifically, the image data of the interface is acquired and preprocessed to obtain the image data to be analyzed, including: The interface is continuously captured according to a preset screenshot frame to obtain a sequence of screenshot images; The screenshot image sequence is subjected to image fusion processing to obtain a fused analysis image; The fused analysis image is preprocessed to obtain the image data to be analyzed.

6. The Web automation testing method based on a two-layer AI architecture of perception and orchestration as described in claim 5, characterized in that, Specifically, the process of performing image fusion processing on the screenshot image sequence to obtain a fused analysis image includes, If the screenshot image sequence is [img_1, img_2, ..., img_P], and i∈[1, P]; The initial fused image is obtained by performing a global equal-weighted average on the screenshot image sequence; Based on the advantage display index, the screenshot image sequence is traversed to find the advantage display area of ​​img_i; The advantageous display areas of all elements in the screenshot image sequence are replaced with the corresponding areas of the initial fused image in ascending order of i, thereby obtaining the fused analysis image; The concept of the Advantage Demonstration Index is explained below: For img_i, if Let be the pixel value of the pixel in the p-th row and q-th column of the two-dimensional matrix of the image image img_i. Advantages Display Index The calculation formula is: ； in, , as well as They are respectively The average, maximum, and minimum pixel values ​​of the 8 neighboring pixels in the two-dimensional matrix of the image img_i.

7. The Web automation testing method based on a two-layer AI architecture of perception and orchestration as described in claim 6, characterized in that, Specifically, the screenshot image sequence is traversed based on the dominant display index to identify the dominant display area of ​​img_i, including, When the value of i is 1, that is, for the first element in the screenshot image sequence, traverse all the pixels of img_1, find the two pixels with the largest advantage display index and denot them as feature_1 and feature_2, generate a straight line feature_12 based on feature_1 and feature_2, then find the pixel with the largest advantage display index feature_3 from the pixels of img_1 other than feature_12, and connect feature_1, feature_2 and feature_3 in sequence to obtain the closed area, which is the advantage display area of ​​img_1; When the value of i is not 1, first calculate the average pixel value of the pixels in the dominant display area of ​​img_i-1 and denote it as Pj_i-1. Traverse all pixels of img_i and randomly find two pixels with a pixel value higher than Pj_i-1 and denote them as feature_4 and feature_5. Generate a straight line feature_45 based on feature_4 and feature_5. Then, find the pixel with the largest dominant display index feature_6 from the pixels in img_i other than feature_45. Connect feature_4, feature_5 and feature_6 in sequence to obtain the closed area, which is the dominant display area of ​​img_i.

8. A Web automation testing device based on a two-layer AI architecture of perception and orchestration, characterized in that, The steps of applying the method according to any one of claims 1-7 include the following: The orchestration layer is configured to receive test tasks, generate and send operation instructions through LLM dynamic programming, call the perception layer, and integrate a state monitoring engine and a schema validator. The perception layer, which is communicatively connected to the orchestration layer, is configured to perform multimodal analysis using a large visual language model (VLM) based on the received image data to be analyzed and structured analysis instructions, and return structured data in a predefined format. The orchestration layer is further configured to generate a new sequence of operation instructions based on the structured data returned by the perception layer through a schema validator, thereby driving the browser execution proxy of the execution layer to complete the test task.

Images