An end-cloud collaborative multi-platform AI vision testing method

Abstract

The application relates to the technical field of software testing and artificial intelligence, and discloses a multi-platform AI visual testing method based on end-cloud cooperation, which comprises the following steps: dividing a testing system into an arrangement reasoning layer, an execution driving layer and a base service layer; the arrangement reasoning layer is arranged on a server end, and all reasoning requests are initiated by a unified large model reasoning agent of the server end; the execution driving layer is arranged across the server end and an edge end, and comprises a first type of execution end located on the server end and a second type of execution end located on the edge end; a step-level cognitive closed loop is performed on each test step of each test case, which comprises the following steps: Step 1, collecting a current screenshot and reporting the server; Step 2, calling a visual large model reasoning to generate a structured operation instruction; Step 3, issuing the execution end to execute and returning the result and a new screenshot; and Step 4, verifying the result and updating the step state.

Description

Technical Field

[0001] This application relates to the field of software automation testing technology, and in particular to a multi-platform AI vision testing method with edge-cloud collaboration. Background Technology

[0002] As enterprises become more digitally savvy, the iteration frequency of core business systems accelerates, leading to a continuous increase in regression testing workload. Traditional GUI automated testing based on frameworks such as Selenium, Playwright, and Appium relies on XPath, CSS selectors, or AccessibilityID to locate interface elements. In scenarios involving page structure changes, dynamic pop-ups, and asynchronous loading, script maintenance costs are high, and failure rates are high, making it difficult to meet the delivery rhythm of high-frequency iterations.

[0003] In recent years, Visual Large Models (VLMs) have been able to perform semantic understanding of screenshots and generate operation decisions, providing a technical path for "visual understanding-driven testing". Some open-source frameworks (such as Midscene.js) have provided atomic operation primitives based on VLM (such as aiTap, aiInput, aiAssert), which can complete visual-driven operations on local pages on a single machine. However, the iOS sandbox mechanism prohibits the installation of third-party inference engines on the device under test or in the test assistance process, making the solution of deploying VLM locally on the device side infeasible on iOS terminals. If only Android uses device-side inference while iOS uses other methods, the decision logic of the same test case cannot be guaranteed to be consistent on different terminals. At the same time, the computing power deployment cost of existing technologies is high and the utilization rate is low, making it difficult for small and medium-sized enterprises to afford the cost of large-scale GPU edge deployment. In addition, existing solutions distribute the complete script to the execution terminal at once for sequential execution. The execution terminal does not request the server to make decisions based on the latest screen status at each step. When the page changes due to dynamic pop-ups, asynchronous loading, permission requests, etc., the script is prone to failure. Therefore, how to reduce the deployment cost of the edge terminal, uniformly schedule multiple execution terminals, and coordinate execution with a step-level cognitive closed-loop protocol is the fundamental problem that this invention aims to solve. Summary of the Invention

[0004] To reduce edge deployment costs and unify the scheduling of multiple execution terminals in collaborative execution using a step-level cognitive closed-loop protocol, this application provides a multi-platform AI vision testing method and system for edge-cloud collaboration, employing the following technical solution: A multi-platform AI vision testing method with edge-cloud collaboration, the method being applied to a testing system including a server and at least one edge device, comprising: The test system is divided into an orchestration and inference layer, an execution-driven layer, and a base service layer. The orchestration and inference layer is deployed on the server side, and all inference requests are initiated through the unified large model inference agent on the server side. The execution-driven layer is deployed across the server side and the edge side, including a first type of execution terminal located on the server side and a second type of execution terminal located on the edge side. For each test step of each test case, a step-level cognitive loop is executed, including: Step 1, capturing a screenshot of the current screen and reporting it to the server; Step 2, calling the visual big model to infer and generate structured operation instructions; Step 3, sending the execution terminal to execute and sending back the results and new screenshots; Step 4, verifying the results and updating the step status. The execution driver layer provides a unified interface to the outside world and automatically translates the unified interface into platform commands for different operating systems; The server maintains the Agent registry and manages each edge terminal; new tasks are allocated according to device availability and Agent capacity, and are automatically queued when a device is occupied. After an Agent disconnects, it is marked as offline after a grace period and the affected tasks are rescheduled.

[0005] Optionally, the first type of execution terminal performs the test through in-process function calls, while the second type of execution terminal performs the test through WebSocket long-connection communication and drives the mobile device via Universal Serial Bus.

[0006] Optionally, the execution of the step-level cognitive closed loop includes a server-controlled mode and an execution-end autonomous mode; The server control mode is as follows: the orchestration and inference layer generates decision instructions for each step and sends them to the execution end; The autonomous execution mode is as follows: the execution end executes autonomously based on local knowledge injection and selector replay, and only requests the server when visual large model inference is required.

[0007] Optionally, the specific process of invoking the visual large model inference in Step 2 includes: Obtain the Top-K candidate operations and their probability distributions from the model output. Calculate the enhanced normalized entropy value H by applying adaptive temperature softening and feature space dispersion penalty to the candidate operation probability distribution. When the enhanced normalized entropy value H exceeds the preset high threshold, direct instruction generation is rejected, and attention-based local region pruning and amplification and secondary inference are triggered until at least one of the following conditions is met: entropy value falls back or the upper limit of recursion depth is reached.

[0008] Optionally, in Step 4, a multimodal confidence fusion model is used to obtain the verification results; The multimodal confidence fusion model includes: Calculate the structural similarity index between the screenshot after execution and the expected reference image; Calculate the CLIP cosine similarity between the screenshot after execution and the semantic text of the steps, as well as the intersection-union ratio of the operation target; The structural similarity index, CLIP cosine similarity, and crossover-union ratio are weighted and fused to obtain the total confidence score. The total confidence level is compared with the dynamic decision threshold. When the total confidence level is less than the dynamic decision threshold, a step-level retry is triggered. The dynamic decision threshold is adaptively updated based on the mean and standard deviation of the historical execution window.

[0009] Optionally, the process of managing each edge endpoint includes: Track the connection status, device list, task capacity, and heartbeat activity of each edge actuator; The Agent registry includes online status, reconnecting status, and offline status. The online status is when the WebSocket connection is active and the heartbeat is normal; the reconnecting status is when the WebSocket connection is interrupted but within the grace period, and the task is paused but not cleared; the offline status is when the grace period has expired and the associated task is marked as a connection interruption. Once a test task is distributed to an Agent and a specific mobile device is locked, other tasks that require the same device are automatically queued, and the queue position is notified to the caller in real time.

[0010] Optionally, the structured message types in the WebSocket communication include screenshot reporting messages, device status messages, execution result messages, operation instruction messages, assertion instruction messages, task allocation messages, task cancellation messages, and checkpoint synchronization messages; In this process, message semantics are decoupled from the transport layer, and the first type of execution end and the second type of execution end reuse the same message semantics through in-process calls and remote network calls, respectively.

[0011] Optionally, when performing step-level cognitive loop at each test step, if a preset checkpoint is hit, the server generates a checkpoint record containing an execution identifier, step number, and status snapshot. After an interruption, execution resumes from the interrupted step based on the checkpoint record, without having to re-execute from the first step of the test case.

[0012] Optionally, the first type of execution end and the second type of execution end reuse the same orchestration protocol and driver abstract interface, and share the inference computing power of the same unified large model inference agent; The server side of the test system is orchestrated using Docker Compose and includes API services, a web frontend, a database, a cache, and object storage.

[0013] Optionally, the orchestration and inference layer maintains bounded policy configurations, including a maximum orchestration round, a list of allowed tools, a screenshot anonymization policy, an audit trail level, and an inference budget cap.

[0014] In summary, this application includes at least one of the following beneficial technical effects: This invention centralizes inference on the server-side with a unified agent, enabling zero-model deployment at the edge, circumventing iOS sandbox limitations, and adopting the same visual decision-making logic across Android / iOS / HarmonyOS / Web. Multiple execution ends share a unified inference backend, triggering inference with screenshot arrival as the event, avoiding distributed GPU deployment and reducing computing costs. Each step is based on the latest screenshot, handling page changes such as dynamic pop-ups and asynchronous loading. Web and Mobile share the same driver abstract interface and protocol semantics, allowing the same use case to be executed across platforms. Agent registry + connection tri-state + device mutex lock achieves fine-grained scheduling, with automatic rescheduling upon heartbeat timeout. Step-level checkpoints enable recovery from breakpoints after interruption, avoiding re-running from the beginning. Attached Figure Description

[0015] Figure 1 This is a flowchart of the multi-platform AI vision testing method in this invention. Detailed Implementation

[0016] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0017] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0018] like Figure 1 As shown in the embodiments of this application, a multi-platform AI vision testing method with edge-cloud collaboration is applied to a testing system including a server and at least one edge device, comprising: Step 1: Divide the test system into an orchestration and inference layer, an execution-driven layer, and a base service layer. The orchestration and inference layer is deployed on the server side, and all inference requests are initiated through the unified large model inference agent on the server side. The execution-driven layer is deployed across the server side and the edge side, including a first type of execution terminal located on the server side and a second type of execution terminal located on the edge side. Step 2: Perform a step-by-step cognitive loop for each test step of each test case, including: Step 1: Collect a screenshot of the current screen and report it to the server; Step 2: Call the visual big model to generate structured operation instructions; Step 3: Send the execution terminal to execute and return the results and new screenshots; Step 4: Verify the results and update the step status; Step 3: The execution driver layer provides a unified interface to the outside world and automatically translates the unified interface into platform commands for different operating systems; Step 4: The server maintains the Agent registry and manages each edge terminal; new tasks are allocated according to device availability and Agent capacity; when a device is occupied, it is automatically queued; after an Agent disconnects, it is marked as offline after a grace period and the affected tasks are rescheduled.

[0019] This invention logically divides the testing system into three layers: the orchestration and inference layer, the execution-driven layer, and the base service layer. The orchestration and inference layer is deployed on the server side, with its core component being the PlatformPlannerService. This service receives step observation information from the execution side, calls the unified large-model inference agent for visual inference, and returns a structured orchestration decision. The orchestration decision includes three types: action (executes an action with specified tools and parameters), complete_step (the current step has passed verification), and fail_step (the current step failed). The orchestration layer maintains bounded policy configurations, including the maximum number of orchestration rounds per step (default 8 rounds), orchestration timeout (default 60 seconds), allowed tool catalog (6 standard tools: ai_navigate, ai_action, ai_input, ai_tap, ai_assert, ai_wait), screenshot anonymization strategy, variable anonymization strategy, audit trail level, and inference budget cap.

[0020] The execution driver layer is deployed across server-side and edge devices, including: a first type of execution end deployed on the server side—the Web executor (driving Chrome / Chromium browsers via Playwright), which communicates with the orchestration layer via in-process function calls, with a typical round-trip latency of less than 10ms; and a second type of execution end deployed on the edge test computer—the mobile agent (communicating with the server via a Socket.IOWebSocket long connection and driving Android / iOS / HarmonyOS devices via USB). This layer exposes a unified interface to the outside world through the driver abstraction layer, including navigation, clicking, input, screenshotting, swiping, key presses, launching applications, assertions, general actions, waiting, and obtaining device information operations. For devices under test with different operating systems, the driver abstraction layer translates the unified interface into platform-specific commands corresponding to that operating system, so that the visual model is unaware of the operating system type of the device under test during inference.

[0021] The base service layer is deployed on the server side, providing basic services such as a unified large model inference agent, agent registry, task scheduling queue, database storage, caching, and message communication; all VLM inference calls must be issued by the orchestration inference layer through the unified large model inference agent of the base layer.

[0022] In addition, the constraints on the edge are: the mobile agent on the edge test computer only contains the communication module, device manager and driver implementation, does not deploy the weight files of the large visual model or the large language model, and does not directly make calls to any model API.

[0023] This invention centralizes inference on the server-side with a unified agent, enabling zero-model deployment at the edge, circumventing iOS sandbox limitations, and adopting the same visual decision-making logic across Android / iOS / HarmonyOS / Web. Multiple execution ends share a unified inference backend, triggering inference with screenshot arrival as the event, avoiding distributed GPU deployment and reducing computing costs. Each step is based on the latest screenshot, handling page changes such as dynamic pop-ups and asynchronous loading. Web and Mobile share the same driver abstract interface and protocol semantics, allowing the same use case to be executed across platforms. Agent registry + connection tri-state + device mutex lock achieves fine-grained scheduling, with automatic rescheduling upon heartbeat timeout. Step-level checkpoints enable recovery from breakpoints after interruption, avoiding re-running from the beginning.

[0024] Taking a single test step as an example, this illustrates the step-level cognitive closed loop, including: Step 1, Screenshot Reporting: The execution end collects a screenshot of the current screen. The Web executor collects the screenshot using the Playwright's page.screenshot() method; the Android end collects the screenshot using the adbscreencap command; the iOS end collects the screenshot using the WebDriverAgent's screenshot interface; and the HarmonyOS end collects the screenshot using the hdcscreenshot command. The screenshot, along with current page observation information (current URL, page title, viewport size), is encapsulated into a decision request and reported to the server.

[0025] Step 2, VLM Inference and Instruction Generation: The server-side PlatformPlannerService receives the decision request and constructs an inference request containing step semantics, historical execution records, and current observations. The unified large-scale model inference agent calls the VLM for visual inference, returning structured operation instructions. These instructions include tool names (e.g., ai_navigate, ai_tap, ai_input, ai_assert, ai_action, ai_wait) and corresponding parameters. Sensitive fields in variables (matching keywords such as password / secret / token / apikey) are automatically masked as a mask string during reporting.

[0026] Specifically, when calling VLM for visual reasoning, a screenshot and the semantic description of the current test step are input into the large visual model. By adjusting the parameters, the top K candidate operation instructions (K≥3) output by the model and their corresponding Softmax probability distributions are obtained. To quantify the model's cognitive certainty regarding the current interface and operations, the enhanced normalized entropy value of the inference probability distribution is calculated. , The probability mass after softening with an adaptive temperature parameter T. This is the feature space dispersion penalty factor between candidate operations. The preset balance coefficient has a value range of [0.2, 2.0]; the default value is 0.5, and the value can be increased for scenarios with extremely high operational precision requirements; the adaptive temperature parameter T is dynamically adjusted according to the global attention entropy during model inference. A is the activation matrix of the last attention map layer of the large model. Let A be the coefficient of variation. When the model's global attention distribution to the image is extremely uneven (i.e., it only focuses on local regions), as the temperature increases, the probability distribution tends to smooth out, forcing the model to rebalance between global and local attention. is the scaling factor, with a value range of [0.1, 2.0] and a default value of 0.8; is the feature space dispersion penalty factor between candidate operations. , It is a dense feature vector mapped to each candidate operation. To prevent small constants from being divided by zero, The larger the value, the greater the divergence in the operation instruction space (coordinates, text, type) among the candidate operations with higher probabilities. Even if the model provides a probability distribution, it is in a high-risk state of "not knowing which direction to operate in". Conversely, if the coordinates of the candidate operations are only a few pixels apart, although the entropy value may be high due to the flat probability, the total entropy value will be quickly reduced after multiplying by the exponential decay term, avoiding unnecessary excessive retries.

[0027] It should be noted that constructing dense feature vectors The specific process includes: (1) For the i-th candidate operation instruction parse its operation type identifier Screen normalized coordinate center point and operation of associated text payload ; (2) By using a pre-trained operation type embedding matrix to... Mapped to a 32-dimensional dense vector ; (3) Through a nonlinear mapping containing 64 radial basis function kernels, the... Expanded into 64-dimensional feature vectors Each kernel function is centered at its corresponding grid anchor point, with a bandwidth of 1 / 16 of the short side length of the screen. (4) Use CLIP text encoder to (If it exists) Compressed into a 416-dimensional semantic embedding vector If no text payload exists, it is filled with a zero vector. (5) Combine the above three subspace vectors , , splicing, and through Unit norm standardization yields the final dense eigenvector. .

[0028] The feature difference distance between two candidate operations is measured using a weighted Euclidean distance, where the operation type subspace is used. The weight is 2.0, and the space subspace is... Weight 1.0, semantic subspace The weight is 1.5; the final feature space dispersion penalty factor ρ is the weighted average of the product of the weighted distance and the corresponding probability mass among all candidate operation pairs.

[0029] The enhanced normalized entropy value ranges from 0 to 1. When H→0, the probability is highly concentrated on a single candidate instruction, indicating that the model is extremely certain. When H→1, the probability distribution is flat, indicating that the model is seriously hesitant, and the current interface may have visual ambiguity or belong to an out-of-distribution scene. The enhanced normalized entropy value H is compared with the preset high threshold and the preset low threshold. If the enhanced normalized entropy value H is less than or equal to the preset low threshold, the instruction corresponding to the Top-1 (first candidate operation instruction) is directly adopted as the final operation instruction, the confidence level is marked as HIGH, and it is quickly sent to the execution end for execution. If the enhanced normalized entropy value H is greater than the preset low threshold and less than or equal to the preset high threshold, the Top-1 instruction is adopted, but the metadata is added to mark the confidence level as MEDIUM, and the entropy value H is also sent to the execution end. After receiving it, the execution end dynamically extends the operation execution interval from the default value in the driver layer and forces an additional screenshot to be added before and after the operation, sacrificing micro-latency for execution reliability under low confidence. If the enhanced normalized entropy value H is greater than the preset high threshold, it is determined that there is serious visual ambiguity in the current global interface, and the direct generation of instructions is rejected. The attention-guided local resampling mechanism is triggered. The attention-guided local resampling mechanism includes: using the attention map built into the visual large model to locate the candidate box of the operation region that the model is most concerned about; cropping and enlarging the original screenshot with the candidate box as the center (e.g., enlarging by 2 times to 448×448 resolution by bicubic interpolation) to obtain a local detail map; and re-feeding the local detail map into the visual large model and executing steps S1021 and S1022 again to calculate the new entropy value. ,like If the value is less than the preset high threshold, the loop will be exited and instructions will be executed according to the first or second level logic. If the recursion depth exceeds the preset upper limit (set to 3 rounds in this embodiment) and the entropy value is always too high, the step will be marked as "environmental abnormality / human intervention required", and the step-level cognitive loop will be exited directly to avoid wasting computing power on invalid operations.

[0030] In this embodiment, the preset low threshold and preset high threshold are set to 0.25 and 0.75, respectively.

[0031] Through the aforementioned entropy self-reflection mechanism, this invention can proactively intervene and optimize when the large model is cognitively uncertain, rather than blindly accepting the model output, thus significantly improving the reliability of test execution.

[0032] Step 3: Command Execution and Result Feedback: The execution end receives the operation command and executes it through the driver abstraction layer. Taking ai_tap as an example: the web end translates it as Playwrightpage.click(); the Android end translates it as adbshellinputtapxy; the iOS end translates it as WDAtap request; and the HarmonyOS end translates it as hdcshelluitestscreenTap. The execution result (passed / failed + time taken + error message) is fed back to the orchestration layer.

[0033] Step 4, Result Verification and Status Update: The orchestration layer determines the step status based on the execution result. If the previous round result was "passed", the "complete_step" decision is returned to proceed to the next step; if it was "failed", the process continues to try within the maximum orchestration rounds; if the maximum number of rounds is exceeded (default 8 rounds), the step is marked as failed.

[0034] The result verification process involves obtaining the screenshot returned by the execution end after execution. Simultaneously, the expected result of this step is read from the server-side database (see reference figure). and the semantic description text of the current step ; The overall confidence score is constructed by weighted fusion of the structural similarity index, CLIP cosine similarity, and crossover-union ratio (CURBR). When the test steps do not include spatial positioning operations, the weight of the CURBR is 0, and the weights of the structural similarity index and CLIP cosine similarity are 0.4 and 0.6, respectively. When the test steps include spatial positioning operations, the weights of the structural similarity index, CLIP cosine similarity, and CURBR are 0.3, 0.5, and 0.2, respectively. The overall confidence score is obtained by weighted summation of the structural similarity index, CLIP cosine similarity, and CURBR with their respective weights.

[0035] Among them, the structural similarity index is calculated and then screenshotted. Reference chart with expected results The value range is [0,1]. This metric is used to capture the macro-consistency of the interface layout and control outlines.

[0036] The semantic-level alignment score is obtained by calling a lightweight multimodal encoder (the CLIP model in this example) and then taking a screenshot after execution. Encoded as visual vectors semantic description text Encoded as text vectors The cosine similarity between the two is calculated, and this metric is used to capture semantic pass / fail.

[0037] The positioning accuracy score is calculated by comparing the expected target bounding box in the operation command with the UI element bounding boxes identified by the lightweight object detection model in the screenshot after the actual execution.

[0038] It should be noted that the structural similarity index and CLIP cosine similarity are calculated using existing algorithms. The intersection-union ratio is the ratio of the intersection region of the expected target bounding box and the UI element box identified by the lightweight object detection model in the screenshot after actual execution to the union region.

[0039] In addition, dynamic threshold determination The algorithm updates adaptively based on the mean and standard deviation of historical execution windows. , This represents the mean of the total confidence level within the historical window. The standard deviation is the window length, which is the result of the most recent N executions (default N=10).

[0040] The step-level closed-loop is implemented by the executeOrchestratedStep function on the execution side. Each round of the loop includes: requesting a decision → executing an action → recording the result → judging the termination condition.

[0041] The driver abstraction layer defines a unified operation interface through the UiDriverBase interface, please refer to the table below.

[0042] The input for VLM inference only includes screenshots and step semantics, and does not accept operating system type fields; operating system differences are completely handled by the driver abstraction layer, ensuring "OS-insensitive" cross-platform consistent decision-making during VLM inference. Both the Web executor (WebUiDriver) and the Mobile executor (MobileUiDriver) implement the UiDriverBase interface, with MobileUiDriver additionally supporting the manual intervention method for test steps that require manual operation.

[0043] Task scheduling is driven by the dispatchQueuedAgentJobs method, which is invoked at the following event trigger points: Agent capacity available (capacity change callback after heartbeat), task completion (triggers the next task dispatch after capacity is released), periodic check (periodic timer not exceeding 36 seconds), and task confirmation timeout (re-queuing if the Agent has not confirmed receipt).

[0044] The task dispatch process is as follows: obtain dispatchable tasks from the queue → check if Run is active → call findBestAgent to find the best Agent (prioritize using the specified Agent, match according to executor preferences, check platform compatibility and device availability) → lock the target device → create a task assignment record → send a job-dispatched message via WebSocket → start the confirmation timeout timer.

[0045] Agent selection strategies support multiple executor preferences: local (prefer local agents), shared (prefer shared agents), and auto (automatic selection); for scheduled and Webhook-triggered tasks, shared agents or CIAgents are preferred.

[0046] When a test task specifies a device serial number, the target device is located via DeviceService. If the device is already locked by another task, false is returned, the task remains in the queue, and its queue position is notified. If the device is available, it is locked, and the locking relationship is recorded. The lock is released by calling the releaseJobDevice method when the task completes or fails. The getBusyDeviceSerialsForDispatch method collects the set of currently locked device serial numbers during dispatch, ensuring that the task is not dispatched to a device that is already occupied.

[0047] The server and the execution end communicate through eight types of structured messages. The message type definition and payload structure are independent of the communication method: the Web executor transmits messages through in-process function calls, and the mobile agent transmits messages through WebSocketJSON. Both use the same message semantics. The table below shows the direction and payload points corresponding to different message types.

[0048] When a test step hits a preset checkpoint condition (such as after every N steps or after a key business node), the server generates a checkpoint record, which includes: execution identifier, test case identifier, step number, current step status, variable snapshot (sensitive fields are masked), and Planner historical round record.

[0049] If execution is interrupted (network disconnection, Agent crash, or timeout), the recovery process is as follows: Agent reconnects or a new Agent receives the task → Server queries the latest checkpoint → If a checkpoint exists, the Agent starts execution from the checkpoint step number, skipping completed steps → If no checkpoint exists, execution starts from step 1. Checkpoint records are cleared after all test cases are completed.

[0050] The orchestration layer maintains the bounded policy configuration of AiOrchestrationConfig to ensure that the step-level closed loop executes within a controllable range. Configuration items include: maximum orchestration rounds per step (default 8), orchestration timeout (default 60,000 milliseconds), allowed tool catalog (6 standard tools), screenshot anonymization policy (choose one from none / metadata_only / redacted), variable anonymization policy (choose one from allowlist / masked), audit trail level (summary / full), and optional inference budget limit (maximum rounds / maximum number of tokens / maximum cost). The orchestration configuration is sent to the execution end along with the task allocation message, and the execution end complies with the configuration constraints in the step-level closed loop.

[0051] This embodiment deploys all server-side components on a single physical machine or virtual machine using Docker Compose, including the API service (NestJS), web frontend (React+Vite), database (PostgreSQL16), cache (Redis7), object storage (RustFS or S3-compatible storage), and WebSocket gateway (Socket.IO). The edge mobile agent is deployed independently on a test computer and communicates with the server-side gateway via WebSocket; this deployment mode is suitable for pilot phases of small and medium-sized enterprises.

[0052] Example 1: Complete Web Mode Process Testers triggered the test case "E-commerce backend - Product management - Add product": 1. The server creates an execution record, and the Web executor launches the Playwright browser; 2. Step-by-step closed-loop: Web executor process reports screenshot → orchestration layer requests VLM inference → returns ai_navigate instruction → Playwright executes page.goto → reports execution result; 3. Navigation verification successful in orchestration layer → Return to the ai_tap command → Playwright executes the click → Verification passed; 4. The orchestration layer returns the ai_input instruction → Playwright executes the input → validation passes; 5. The orchestration layer returns the ai_assert instruction → VLM inference verification → assertion passed; 6. Once all steps are completed, an execution report will be generated.

[0053] The entire process is model-free at the edge, and inference requests are initiated through a unified proxy.

[0054] Example 2: Complete Process of Mobile Mode The test engineer's computer is running a mobile agent, with one Android and one iOS device connected via USB: 1. After the mobile agent starts, it scans for USB devices and registers them in the Agent registry based on device status messages; 2. Both devices are marked as online, and two execution tasks are created on the server side; 3. Android device executes step-by-step closed loop: adb screenshot reporting → VLM inference (enhanced normalized entropy value determines success of this step) → returns ai_tap → driver translates to adbshellinputtap → verification passed (total confidence = 0.863, greater than dynamic judgment threshold) =0.850, verification passed); 4. Step 5: Hit the checkpoint and save the checkpoint record; 5. Step 8: Network intermittent interruption → Agent status set to reconnecting → Task paused; After 6.5 seconds, the network recovers → the agent reconnects → the status returns to online → execution resumes from checkpoint step 5; 7. iOS devices also bypass sandbox restrictions by using server-side inference; 8. Both devices share the same VLM inference backend, and the evidence format is consistent.

[0055] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A multi-platform AI vision testing method with edge-cloud collaboration, characterized in that, The method is applied to a testing system comprising a server and at least one edge terminal, including: The test system is divided into an orchestration and inference layer, an execution-driven layer, and a base service layer. The orchestration and inference layer is deployed on the server side, and all inference requests are initiated through the unified large model inference agent on the server side. The execution-driven layer is deployed across the server side and the edge side, including a first type of execution terminal located on the server side and a second type of execution terminal located on the edge side. For each test step of each test case, a step-level cognitive loop is executed, including: Step 1, capturing a screenshot of the current screen and reporting it to the server; Step 2, calling the visual big model to infer and generate structured operation instructions; Step 3, sending the execution terminal to execute and sending back the results and new screenshots; Step 4, verifying the results and updating the step status. The execution driver layer provides a unified interface to the outside world and automatically translates the unified interface into platform commands for different operating systems; The server maintains the Agent registry and manages each edge terminal; new tasks are allocated according to device availability and Agent capacity, and are automatically queued when a device is occupied. After an Agent disconnects, it is marked as offline after a grace period and the affected tasks are rescheduled.

2. The multi-platform AI vision testing method with edge-cloud collaboration according to claim 1, characterized in that, The first type of execution terminal performs the test through in-process function calls, while the second type of execution terminal performs the test through WebSocket long-connection communication and drives the mobile device via Universal Serial Bus.

3. The multi-platform AI vision testing method with edge-cloud collaboration according to claim 2, characterized in that, The execution of the step-level cognitive closed loop includes a server-controlled mode and an execution-end autonomous mode; The server control mode is as follows: the orchestration and inference layer generates decision instructions for each step and sends them to the execution end; The autonomous execution mode is as follows: the execution end executes autonomously based on local knowledge injection and selector replay, and only requests the server when visual large model inference is required.

4. The multi-platform AI vision testing method with edge-cloud collaboration according to claim 1, characterized in that, The specific process of invoking the large visual model inference in Step 2 includes: Obtain the Top-K candidate operations and their probability distributions from the model output. Calculate the enhanced normalized entropy value H by applying adaptive temperature softening and feature space dispersion penalty to the candidate operation probability distribution. When the enhanced normalized entropy value H exceeds the preset high threshold, direct instruction generation is rejected, and attention-based local region pruning and amplification and secondary inference are triggered until at least one of the following conditions is met: entropy value falls back or the upper limit of recursion depth is reached.

5. The multi-platform AI vision testing method with edge-cloud collaboration according to claim 1, characterized in that, In Step 4, a multimodal confidence fusion model is used to obtain the verification results; The multimodal confidence fusion model includes: Calculate the structural similarity index between the screenshot after execution and the expected reference image; Calculate the CLIP cosine similarity between the screenshot after execution and the semantic text of the steps, as well as the intersection-union ratio of the operation target; The structural similarity index, CLIP cosine similarity, and crossover-union ratio are weighted and fused to obtain the total confidence score. The total confidence level is compared with the dynamic decision threshold. When the total confidence level is less than the dynamic decision threshold, a step-level retry is triggered. The dynamic decision threshold is adaptively updated based on the mean and standard deviation of the historical execution window.

6. The multi-platform AI vision testing method with edge-cloud collaboration according to claim 1, characterized in that, The process of managing each edge endpoint includes: Track the connection status, device list, task capacity, and heartbeat activity of each edge actuator; The Agent registry includes online status, reconnecting status, and offline status. The online status is when the WebSocket connection is active and the heartbeat is normal; the reconnecting status is when the WebSocket connection is interrupted but within the grace period, and the task is paused but not cleared; the offline status is when the grace period has expired and the associated task is marked as a connection interruption. Once a test task is distributed to an Agent and a specific mobile device is locked, other tasks that require the same device are automatically queued, and the queue position is notified to the caller in real time.

7. The multi-platform AI vision testing method with edge-cloud collaboration according to claim 2, characterized in that, The structured message types in the WebSocket communication include screenshot reporting messages, device status messages, execution result messages, operation instruction messages, assertion instruction messages, task allocation messages, task cancellation messages, and checkpoint synchronization messages; In this process, message semantics are decoupled from the transport layer, and the first type of execution end and the second type of execution end reuse the same message semantics through in-process calls and remote network calls, respectively.

8. The multi-platform AI vision testing method with edge-cloud collaboration according to claim 2, characterized in that, When executing the step-level cognitive loop in each test step, if a preset checkpoint is hit, the server generates a checkpoint record containing the execution identifier, step number, and status snapshot. After an interruption, execution resumes from the interrupted step based on the checkpoint record, without having to re-execute from the first step of the test case.

9. The multi-platform AI vision testing method with edge-cloud collaboration according to claim 1, characterized in that, The first type of execution end and the second type of execution end reuse the same orchestration protocol and driver abstract interface, and share the inference computing power of the same unified large model inference agent; The server side of the test system is orchestrated using Docker Compose and includes API services, a web frontend, a database, a cache, and object storage.

10. The multi-platform AI vision testing method with edge-cloud collaboration according to claim 1, characterized in that, The orchestration and inference layer maintains bounded policy configurations, including the maximum orchestration rounds, allowed tool catalogs, screenshot anonymization policies, audit trail levels, and inference budget caps.

Images