A graphical interface interaction error correction method and system based on asynchronous perception and closed-loop verification and a medium

Abstract

This invention discloses a graphical user interface (GUI) interaction error correction method, system, and medium based on asynchronous perception and closed-loop verification. The method includes: receiving image-instruction pairs in a dynamic environment and extracting multimodal features from the observed image; calculating the similarity between the current features and the multimodal interaction constraint dictionary through a cross-modal interference pre-perception module; if an anomaly is detected, immediately pausing the main task to block contextual interference; routing the contaminated image and the matching recovery instruction to an isolated trial and closed-loop error correction module to generate a spatial recovery action; performing state verification based on pixel-level rendering differences to objectively verify whether the anomaly has been completely cleared; if not cleared, performing spatial iterative reflection based on the failure coordinates until verification is passed, and then sending the clean observation state back to the main strategy. This method effectively alleviates the spatiotemporal unpredictability and visual alignment errors caused by GUI anomalies in a plug-and-play manner, significantly improving the success rate of the agent in long-range tasks.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, specifically to a graphical interface interaction error correction method, system, and medium based on asynchronous perception and closed-loop verification. Background Technology

[0002] With the rapid development of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), graphical user interface (GUI) agents have gradually evolved into intelligent assistants capable of automating complex digital tasks. Recent visual GUI agents (such as CogAgent, AppAgent, and UI-TARS) have abandoned the traditional parsing method that relies on backend code (such as DOM trees), directly mapping the raw screen pixel visual input to precise spatial coordinates, thereby achieving cross-platform automated operation.

[0003] However, real-world digital environments are inherently dynamic and contain numerous unpredictable user interface (UI) anomalies. When visual GUI agents are deployed in these real-world environments, they expose significant out-of-distribution (OOD) vulnerabilities. Existing technologies face two main challenges in handling dynamic environments: First, there is the unpredictability of time and space. Seven types of UI anomalies, such as action failures, login pages, CAPTCHA pages, pop-up ads, cookie pop-ups, page loading delays, and network disconnections, often appear at random times, in random sizes, and in random locations. This randomness makes traditional data-scaling methods, which attempt to cover edge cases through large-scale model retraining, extremely inefficient.

[0004] Secondly, contextual interference: when an anomaly occurs, forcing the agent's main policy model to process prominent visual distractions simultaneously with the main task instructions will severely disrupt its visual attention mechanism. This concurrent processing leads to serious spatial visual alignment errors and coordinate drift, causing irreversible error accumulation in long-range task trajectories, ultimately resulting in task failure. Summary of the Invention

[0005] This invention proposes a graphical user interface interactive error correction method, system, and medium based on asynchronous perception and closed-loop verification to solve the problems mentioned in the background.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A graphical user interface interactive error correction method based on asynchronous perception and closed-loop verification, comprising the following steps: Receive image-instruction pairs in a dynamic environment and extract multimodal features from the observed images; Based on the extracted multimodal features, calculate the similarity between the current feature and the multimodal interaction constraint dictionary; If the maximum similarity score obtained is greater than or equal to the preset interception threshold, it is determined that UI abnormal interception is triggered. At this time, the main task strategy of the visual GUI agent is immediately suspended to seize system control. Extract the recovery instructions corresponding to the successfully matched anomaly template; Initialize an independent local sandbox context to shield the global history context of the main task strategy; then, in the isolated state of the local sandbox context, generate and execute a space restoration action for the current UI anomaly according to the restoration instruction; Perform state verification based on pixel-level rendering differences to objectively verify whether the anomaly has been completely cleared. If it has not been cleared, perform spatial iterative reflection based on the failure coordinates until the verification is passed, and then send the clean observation state back to the main strategy.

[0007] Preferably, S1 specifically includes: Get the current screen observation image in the continuous task data stream. and natural language commands The screen observation image is extracted using a text feature extractor and a visual feature extractor, respectively. Real-time text semantic collection Visual perception hash encoding .

[0008] Preferably, S2 specifically includes: The extracted text semantic set Visual perception hash encoding The input is fed into the cross-modal interference pre-sensing module to calculate the current screen observation image. With a pre-built multimodal interaction constraint dictionary Various exception templates Multimodal fusion similarity score .

[0009] Preferably, the multimodal interaction constraint dictionary is represented as a finite set. ,in The total number of preset anomaly categories, Describes the first element in this finite set. There are 10 historical exception templates, and each exception template... Each consists of a triplet Explicit definition: This represents a bimodal anomaly signature, used as a retrieval key for deterministic anomaly identification. The signature contains a pre-defined set of sparse semantic keywords. and pre-extracted local visual perception hash vector ; This indicates a recovery command, which provides context-isolated prompt text and offers spatial operation guidance for clearing the corresponding exceptions within the auxiliary recovery strategy of the isolation probing and closed-loop error correction module. This represents a backoff constraint, used to define the maximum number of autonomous recovery steps or the upper limit of time, serving as an algorithmic circuit breaker boundary to prevent the agent from falling into an infinite loop.

[0010] Preferably, the formula for calculating the multimodal fusion similarity score is as follows:

[0011] in, This represents a sentence encoder. The text representation for the current screen. For abnormal templates Semantic keyword representation; Used to calculate the semantic cosine similarity between the two; For the current visual perception hash encoding, For abnormal templates Visually perceived hash vector; Used to calculate the visual similarity between the two based on Hamming distance; To balance the hyperparameters of text modality and visual modality weights, and satisfy... .

[0012] Preferred, S6 specifically includes: A structural similarity algorithm is used to perform state verification based on pixel-level rendering differences on the screen observation images before and after the execution of the spatial restoration action; If the verification fails, the spatial coordinates of the failed execution are extracted to construct a prompt message containing spatial penalties, and this message is appended to the local sandbox context to guide the generation of the next spatial recovery action until the verification passes or the maximum number of iterations is reached. If the verification passes, it is determined that the current UI anomaly has been completely cleared. Then, the cleaned screen observation image is output and returned to the main task strategy. Finally, the suspended state is lifted to resume the execution of the original main task.

[0013] Preferably, the physical verification and spatial reflection process is as follows: Execute the first step within the isolated probing and closed-loop error correction module. Subspace recovery action Then, the environment will return the current candidate state image. At this point, the candidate state image is calculated. Images observed on contaminated screens Physical structural differences between The formula is as follows:

[0014] like ( If the preset structural mutation threshold is used, it is determined that the current interface rendering has not undergone a fundamental change, i.e., the spatial restoration action... Failure occurs at the physical level; at this point, the isolated probing and closed-loop error correction module will extract the spatial two-dimensional coordinates of the current failed action and normalize them as follows: The format is added as a space penalty to the sandbox context, forcing the auxiliary recovery strategy to be generated in the first... Sub-action Avoid the penalty area where the failure coordinate point is located; like If the space restoration action has successfully destroyed or cleared the local rendering layer where the UI anomaly is located, then the physical verification is passed.

[0015] A graphical user interface-based error correction system based on asynchronous sensing and closed-loop verification includes: The system includes: The multimodal interaction constraint dictionary module is used to build and dynamically maintain the multimodal interaction constraint dictionary, and explicitly define the triplet structure of each exception template; The real-time feature extraction module is used to extract the multimodal text semantic set and visual perception hash code of the screen observation image at each task time step; The cross-modal interference pre-perception module is used to perform nearest neighbor search by fusing similarity metrics. When a UI anomaly is matched, the main task strategy of the large model is forcibly suspended to achieve physical blocking of the perception layer. The isolated probing and closed-loop error correction module is used to create a virtual execution environment with context isolation characteristics, perform spatial recovery actions for UI anomalies, and use the similarity of the screen structure before and after execution as a verification constraint to perform spatial coordinate penalties and iterative reflection. The Clean Observation Status Feedback Module is used to output a clean screen observation status and send it back to the main task strategy after the status verification is successful, so as to restore the original task flow.

[0016] Preferably, the cross-modal interference pre-sensing module configures an interception threshold. and dual-mode equilibrium coefficient This enables physical preemption interception of contaminated images before they enter the core neural network of the agent, thereby completely eliminating contextual attention pollution caused by abnormal interference.

[0017] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0018] As can be seen from the above technical solution, this invention provides a graphical interface interactive error correction method based on asynchronous perception and closed-loop verification. Compared with the prior art, this invention has the following advantages: Through interactive data flow decoupling and asynchronous error correction mechanisms, this invention can effectively block contextual interference without requiring any parameter fine-tuning of the basic large model; in authoritative Robust and EnvDistraction benchmark tests, this invention significantly improves the spatial alignment accuracy of the agent when facing severe visual interference, and relatively improves the success rate of long-range end-to-end tasks by 4.1 times, significantly enhancing generalization ability and robustness. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification according to the present invention.

[0020] Figure 2 This is a diagram illustrating the overall framework of the interactive data flow decoupling and asynchronous error correction mechanism of this invention.

[0021] Figure 3 This is a visual comparison chart of the recovery effects of the present invention for different visual disturbances in a multi-stage task. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0023] like Figure 1 As shown in the figure, a graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification in this embodiment includes the following steps: S1: Real-time extraction of multimodal features; Receives image-instruction pairs from a continuous data stream; in an ideal Markov dynamic environment without UI anomalies, visual observations directly correspond to operable states (i.e., ...). However, in real-world environments, visual distractions can disrupt rendering functions, leading to contaminated visual observations. (Destruction), its mathematical model is expressed as:

[0024] in, This represents a matrix of visual distractions (such as pop-up ads). To represent the binary matrix of the spatial masking of interfering objects, This represents the element-wise Hadamard product.

[0025] To identify this damage, the system observes the image from the current screen. Extracting semantic sets from text using OCR And use a sentence encoder to obtain its text representation. Simultaneously, visual feature encoding is extracted using a perceptual hashing algorithm. .

[0026] S2: Prior matching and pre-emptive anomaly interception; Constructing a multimodal interaction constraint dictionary ,Include An exception template (e.g., set an exception template) Each template is composed of... The extracted features are input into the cross-modal interference pre-perception module and matched with the template features in the multimodal interaction constraint dictionary. The multimodal fusion similarity score formula is as follows:

[0027] In this embodiment, the multimodal fusion weights The preferred value is 0.7. This represents the calculation of semantic cosine similarity. This represents visual similarity after Hamming distance normalization.

[0028] S3: Preemptive control and context isolation; If the maximum score matched satisfies (Interception threshold) (Preferred value: 0.52) The cross-modal interference pre-sensing module immediately intercepts the contaminated image. The main task is suspended to prevent historical data corruption. The system extracts the recovery instructions corresponding to the template. (For example, "click the close button in the upper right corner"), and send them to the isolated probing and closed-loop error correction module.

[0029] S4: Execution of space restoration actions within the isolated probing and closed-loop error correction module; Within the isolated probing and closed-loop error correction module, the local context is initialized. The auxiliary recovery strategy generates spatial actions based on this isolation context. That is, find the coordinates of the close button on the screen and click it.

[0030] S5: State verification and spatial reflection iteration based on pixel-level rendering differences; Execute action Candidate state images are then obtained. Calculate the structural similarity difference between it and the contaminated image:

[0031] To accommodate visual distractions of varying sizes and types in real-world digital environments, this embodiment abandons a single, fixed global threshold and instead employs a threshold bound to the multimodal interaction constraint dictionary template. Adaptive structural mutation threshold The adaptive structural mutation threshold The offline calibration strategy and mapping logic are as follows: (1) Threshold calibration strategy: Based on the offline-constructed historical abnormal transition verification set, the statistical distribution of the differences in view structure before and after different types of interference are cleared is statistically analyzed; the objective function is optimized by grid search to achieve a balance between suppressing false positives (that is, misjudging normal interaction micro-movements such as button highlighting and cursor blinking as abnormalities that have been cleared) and false negatives (that is, valid clicks occur but the interface exit is not correctly perceived), thereby anchoring the optimal mutation boundary for each type of template.

[0032] (2) Dynamic classification mapping: When the multimodal interaction constraint dictionary matches the current template When used for full-screen or large-area visual interference (such as page transitions, full-screen ad overlays, or login blocking pages), due to its spatial overlay... With high coverage and significant abrupt changes in rendering topology before and after cleanup, the system automatically assigns weights and matches a high structural mutation threshold (preferred in this embodiment). This enhances immunity to background dynamic loading noise; when the matched template When it is a localized or small-area visual disturbance (such as a floating cookie authorization prompt, a floating window in the lower right corner, or a local verification code box), it affects the overall rendering function. The changes are extremely localized, and the system automatically assigns weights and matches a more sensitive structural mutation threshold (preferred in this embodiment). This ensures the accurate capture of extremely subtle physical states.

[0033] During the sandbox validation phase, if the calculated physical structure differences satisfy... This indicates that the physical rendering layer of the image has not undergone any fundamental changes corresponding to the expected abnormal cleanup, and the spatial restoration action is therefore determined. The failure occurs at the physical level (e.g., a click lands in an invalid blank area or fails to trigger a close response). In this case, the isolated probing and closed-loop error correction module automatically extracts and normalizes the spatial two-dimensional coordinates of the currently failed execution point. The format is added as a space penalty to the local sandbox context. In the next decision time step, the model generates the spatially offset corrected first decision time step based on the context infused with objective physical failure facts and penalty coordinates. Sub-action The process involves a continuous iterative cycle of trial and error, feedback and reflection, until the physical verification constraints are met or the preset maximum number of iterations is reached. (To prevent the system from falling into a state of token exhaustion and infinite loop, this embodiment is preferably...) If the number of iterations... If the validation still fails, the system will automatically trigger the circuit breaker and the specific backoff constraint defined in the current template. The processing authority will be transferred to manual intervention or a system-level reset interface.

[0034] S6: Pure observation status feedback; Once verification satisfies That is, confirm the rendering function. Normal operation has been restored; the UI anomaly has been physically cleared. The clean observation state feedback module outputs clean observation images. Release the main strategy suspension and restore the original execution flow of the long line-of-sight task.

[0035] This invention also discloses a graphical user interface interactive error correction system based on asynchronous perception and closed-loop verification, comprising: The multimodal interaction constraint dictionary module is used to build and dynamically maintain the multimodal interaction constraint dictionary; The real-time feature extraction module is used to output text semantics and visual encoding through pre-trained OCR and hashing algorithms; The cross-modal interference pre-sensing module is used to calculate the fusion similarity metric and intercept the data stream and suspend the main strategy when an anomaly is detected. An isolated probing and closed-loop error correction module is used to generate recovery actions in an isolated environment and to perform state verification and spatial reflection based on pixel-level rendering differences of actions by using SSIM to calculate structural differences. The clean observation status feedback module is used to return the clean image that has passed the verification to the main model.

[0036] The following reference Figure 2 The system architecture and the decoupled execution mechanism of the interactive data flow of this invention will be further described in detail. Figure 2 As shown, the overall execution framework of this invention is mainly composed of a bottom perception layer and a top error correction layer, both physically and logically: At the bottom perception and recognition stage, the real-time feature extraction module receives an interactive data stream containing natural language commands and real-time screen screenshots. The cross-modal interference pre-perception module first uses Optical Character Recognition (OCR) technology to extract the text semantic set in the current image, and simultaneously extracts visual feature codes by calculating local sensory hashes (pHash); subsequently, the aforementioned multimodal features are combined with… Figure 2 The pre-built "multimodal interaction constraint dictionary" on the left is used for similarity measurement and matching. The multimodal interaction constraint dictionary stores in a structured manner various common abnormal screen features (such as network interruption page, pop-up blocker page, etc.), corresponding abnormal keyword text (such as "not connected to the Internet", "check network cable", "reconnect to Wi-Fi network", etc.) and matching recovery prompt instructions (such as "network abnormality detected, please transfer to human" or "click the close button").

[0037] During the top-level isolated probing and closed-loop error correction phase, when the similarity matching score calculated by the cross-modal interference pre-sensing module exceeds the preset interception threshold, the system immediately seizes control and blocks the connection between the main task and the currently contaminated observation screen. For example... Figure 2 As shown in the top left corner, the global historical context of the original main task strategy (e.g., task description: "Check the weather") and the contaminated image are physically isolated and forcibly suspended using dashed boxes (marked with a red pre-blocking indicator) to prevent erroneous data from contaminating the agent's core neural network. Subsequently, the system routes the current input image and the "recovery hint" retrieved from the multimodal interaction constraint dictionary to... Figure 2 Within the green "Isolation Probing and Closed-Loop Error Correction Module" at the top, the system initializes an independent local sandbox context, calls the underlying graphical user interface agent to generate spatial actions to respond to the current anomaly, and then... Figure 2 The center's "Reflection Module" performs structural similarity (SSIM) checks on the screen images before and after the action is executed, based on pixel-level rendering differences.

[0038] like Figure 2 As shown in the "Select Module" path at the top right, this closed-loop verification and spatial reflection mechanism has intelligent branch routing capabilities: If the action verification is successful (i.e., it is physically confirmed that the UI interference has been completely cleared), the selection module executes the "Independent Solution" path, transferring control of the purified screen back to the main agent on the left through the clean observation state feedback module, releasing the suspended state and resuming the normal long-range task; if the interference is still not cleared after multiple reflections and retries, or if the maximum number of steps and time constraints preset in the multimodal interaction constraint dictionary are triggered, the selection module automatically executes the "Transfer to Human" fallback branch, safely transferring the interface with the current status and prompts to the human expert for takeover. This decoupling mechanism enables the asynchronous and independent operation of the main task's long-range execution and the underlying sudden interference investigation.

[0039] It should be further noted that, based on the same inventive concept, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media.

[0040] To further verify the real technical effect and robustness of the graphical interface interaction error correction method and system based on asynchronous perception and closed-loop verification proposed in this invention, the applicant conducted systematic comparative experiments and data verification based on authoritative industry test sets for complex dynamic digital environments (including Test Set 1, which focuses on evaluating contextual interference of single-step atomic operations, and Test Set 2, which focuses on evaluating long-range spatiotemporal unpredictability and anomaly accumulation).

[0041] First, regarding the effectiveness in resisting contextual interference and improving visual alignment accuracy: In test set 1, which includes 1198 single-step high-interference tasks, existing mainstream end-to-end visual agents without the present invention (such as baseline model Y) suffer from severe contamination of their neural network attention when faced with sudden visual interference from adjacent target elements, resulting in a localization accuracy of only 7.40%. The localization accuracy of existing general-purpose large models is generally below 7%. However, by combining the interactive data flow decoupling and asynchronous error correction mechanism proposed in this invention, and through the physical blocking of the cross-modal interference pre-perception module at the perception layer, interference is shielded from entering the large model. This allows the same baseline model to achieve a localization accuracy of 68.77% without any modification to the underlying parameters (training-free), a significant improvement of over 61% in absolute value. When this performance is generalized to another visual architecture (baseline model X), the localization accuracy also steadily increases from 52.73% of the baseline to 62.15%. Specific experimental comparison data are shown in Table 1.

[0042] Table 1 Comparison of Spatial Positioning Accuracy of Various Technical Solutions in Complex Interference Environments Experimental groups and model configuration Spatial positioning accuracy (%) Existing general-purpose visual large model A 0.00 Existing general code programming model B 0.15 Existing general multimodal large model C 3.02 Existing dedicated interface interaction model D 6.64 Dedicated interface interaction baseline model X (not using this invention) 52.73 Baseline Model X + Invention Asynchronous Error Correction System 62.15 End-to-end interface interaction baseline model Y (not using this invention) 7.40 Baseline model Y + Invention asynchronous error correction system 68.77 The following reference Figure 3 Through specific visual comparisons of application scenarios, the single-step alignment and error correction effect of this invention under sudden visual interference is explained in detail. For example... Figure 3 As shown, in a typical automated operation scenario of an e-commerce shopping platform, the agent is performing a long-view task of browsing or selecting products. At this moment, a bright and clearly directional limited-time promotional advertisement pop-up suddenly appears in the center of the interface (as shown in the figure, it includes the prompt "FLASHSALE:SAVE40%OFF!" and a striking orange "SHOPNOW!" shopping button).

[0043] When using existing end-to-end visual agent methods that do not integrate this invention (such as...) Figure 3 The output of the existing method (indicated by the thick green arrow below) suffers from severe spatial visual alignment errors and coordinate drift due to the lack of physical blocking and context isolation mechanisms within the system's perception layer. The main policy model's attention mechanism is heavily contaminated by large, high-contrast distractions in the center of the screen. This results in a lack of physical blocking and context isolation mechanisms within the system. Consequently, the model incorrectly maps the next action to the "SHOPNOW!" button in the pop-up ad, forcibly redirecting the page to an unrelated ad webpage. This interrupts the original shopping task and causes irreversible error accumulation.

[0044] Conversely, when using the error correction system based on asynchronous sensing and closed-loop verification proposed in this invention (such as...) Figure 3 The blue arrow at the top indicates the "output result of the method of this invention" (the cross-modal interference pre-sensing module at the bottom identifies the pop-up's feature matching the advertising interference category in the multimodal interaction constraint dictionary in real time, instantly triggering pre-interception and forcibly suspending the main task strategy. Subsequently, the contaminated screen is routed to the isolated probing and closed-loop error correction module. The system performs precise spatial positioning in a virtual execution environment that shields historical context, accurately identifying the tiny circle close icon in the upper right corner of the advertising pop-up and accurately outputting the spatial coordinate command "Click to close the advertisement". Only after the closing action is completed and SSIM verification confirms that the pop-up rendering layer has been completely destroyed at the physical level is the clean webpage environment observation state sent back to the main task strategy, thus completely eliminating the coordinate offset and accidental touch risk induced by the highlighted interference, ensuring accurate spatial alignment and continuity of long-distance tasks in complex dynamic interactive environments.

[0045] Second, regarding the effectiveness in overcoming the unpredictability of time and space and improving the success rate of long-range missions: In the long-range test set 2, which includes seven categories of dynamic interaction anomalies and trajectories ranging from several to over ten steps, existing general-purpose models, even with a certain accuracy in text reasoning actions, generally suffer from a 0% end-to-end success rate in long-range tasks due to the lack of effective external physical verification and anomaly removal mechanisms. This is caused by coordinate drift and irreversible error accumulation when faced with randomly triggered interface mutations. In contrast, this invention, relying on the pre-interception of a multimodal interaction constraint dictionary and the collaborative work of isolated probing and closed-loop error correction modules, successfully prevents the penetration of erroneous states into the global historical trajectory. Experimental results show that this invention relatively improves the end-to-end task success rate of baseline model X from 2.14% to 8.85% (an improvement of over 4.1 times); and significantly increases the end-to-end task success rate of baseline model Y from 11.57% to 17.07%, achieving excellent levels in both action type accuracy (reaching 53.51%) and spatial coordinate hit rate (reaching 76.25%). Specific experimental comparison data are shown in Table 2.

[0046] Table 2. Comparison of the overall robustness of various technical solutions in long-range dynamic anomaly tasks. Experimental groups and model configuration Action type accuracy (%) Spatial positioning accuracy (%) End-to-end task success rate (%) Existing large-scale general model E 48.54 0.98 0.00 Existing commercial closed-source multimodal model F 45.15 5.13 0.00 Existing commercial closed-source multimodal model G 55.07 1.95 0.00 Dedicated interface interaction baseline model X (not using this invention) 23.48 44.35 2.14 Baseline Model X + Invention Asynchronous Error Correction System 26.15 55.20 8.85 End-to-end interface interaction baseline model Y (not using this invention) 47.06 70.70 11.57 Baseline model Y + Invention asynchronous error correction system 53.51 76.25 17.07 In summary, this invention, through interactive data flow decoupling and asynchronous error correction mechanisms, allows the system to effectively block contextual interference without any parameter fine-tuning of the basic large model (training-free). In authoritative Robust and EnvDistraction benchmark tests, this invention significantly improves the spatial alignment accuracy of the agent when facing severe visual interference and relatively increases the success rate of long-range end-to-end tasks by 4.1 times, significantly enhancing generalization ability and robustness.

[0047] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).

[0048] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0049] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0050] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification, characterized in that, Includes the following steps: Receive image-instruction pairs in a dynamic environment and extract multimodal features from the observed images; Based on the extracted multimodal features, calculate the similarity between the current feature and the multimodal interaction constraint dictionary; If the maximum similarity score obtained is greater than or equal to the preset interception threshold, it is determined that UI abnormal interception is triggered. At this time, the main task strategy of the visual GUI agent is immediately suspended to seize system control. Extract the recovery instructions corresponding to the successfully matched anomaly template; Initialize an independent local sandbox context to shield the global history context of the main task strategy; then, in the isolated state of the local sandbox context, generate and execute a space restoration action for the current UI anomaly according to the restoration instruction; Perform state verification based on pixel-level rendering differences to objectively verify whether the anomaly has been completely cleared. If it has not been cleared, perform spatial iterative reflection based on the failure coordinates until the verification is passed, and then send the clean observation state back to the main strategy.

2. The graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification according to claim 1, characterized in that: The process of receiving image-instruction pairs in a dynamic environment and extracting multimodal features from the observed images specifically includes: Get the current screen observation image in the continuous task data stream. and natural language commands The screen observation image is extracted using a text feature extractor and a visual feature extractor, respectively. Real-time text semantic collection Visual perception hash encoding .

3. The graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification according to claim 2, characterized in that: The step of calculating the similarity between the current feature and the multimodal interaction constraint dictionary based on the extracted multimodal features specifically includes: The extracted text semantic set Visual perception hash encoding The input is fed into the cross-modal interference pre-sensing module to calculate the current screen observation image. With a pre-built multimodal interaction constraint dictionary Various exception templates Multimodal fusion similarity score .

4. The graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification according to claim 3, characterized in that: The multimodal interaction constraint dictionary is represented as a finite set. ,in The total number of preset anomaly categories, Describe the first in this finite set There are 10 historical exception templates, and each exception template... Each consists of a triplet Explicit definition: This represents a bimodal anomaly signature, used as a retrieval key for deterministic anomaly identification. The signature contains a pre-defined set of sparse semantic keywords. and pre-extracted local visual perception hash vector ; This indicates a recovery command, which provides context-isolated prompt text and offers spatial operation guidance for clearing the corresponding exceptions within the auxiliary recovery strategy of the isolation probing and closed-loop error correction module. This represents a backoff constraint, used to define the maximum number of autonomous recovery steps or the upper limit of time, serving as an algorithmic circuit breaker boundary to prevent the agent from falling into an infinite loop.

5. The graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification according to claim 4, characterized in that: The formula for calculating the multimodal fusion similarity score is as follows: in, This represents a sentence encoder. The text representation for the current screen. For abnormal templates Semantic keyword representation; Used to calculate the semantic cosine similarity between the two; For the current visual perception hash encoding, For abnormal templates Visually perceived hash vector; Used to calculate the visual similarity between the two based on Hamming distance; To balance the hyperparameters of text modality and visual modality weights, and satisfy... .

6. The graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification according to claim 5, characterized in that: The execution of pixel-level rendering difference-based state verification objectively verifies whether anomalies have been completely cleared. If not, spatial iterative reflection is performed using the failure coordinates until verification is successful, at which point the clean observation state is sent back to the main strategy. This specifically includes: A structural similarity algorithm is used to perform state verification based on pixel-level rendering differences on the screen observation images before and after the execution of the spatial restoration action; If the verification fails, the spatial coordinates of the failed execution are extracted to construct a prompt message containing spatial penalties, and this message is appended to the local sandbox context to guide the generation of the next spatial recovery action until the verification passes or the maximum number of iterations is reached. If the verification passes, it is determined that the current UI anomaly has been completely cleared. Then, the cleaned screen observation image is output and returned to the main task strategy. Finally, the suspended state is lifted to resume the execution of the original main task.

7. The graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification according to claim 6, characterized in that: The process of physical verification and spatial reflection is as follows: Execute the first step within the isolated probing and closed-loop error correction module. Subspace recovery action Then, the environment will return the current candidate state image. At this point, the candidate state image is calculated. Images observed on contaminated screens Physical structural differences between The formula is as follows: like ( If the preset structural mutation threshold is used, it is determined that the current interface rendering has not undergone a fundamental change, i.e., the spatial restoration action... Failure occurs at the physical level; at this point, the isolated probing and closed-loop error correction module will extract the spatial two-dimensional coordinates of the current failed action and normalize them as follows: The format is added as a space penalty to the sandbox context, forcing the auxiliary recovery strategy to be generated in the first... Sub-action Avoid the penalty area where the failure coordinate point is located; like If the space restoration action has successfully destroyed or cleared the local rendering layer where the UI anomaly is located, then the physical verification is passed.

8. A graphical user interface (GUI) interactive error correction system based on asynchronous perception and closed-loop verification, employing the GUI interactive error correction method based on asynchronous perception and closed-loop verification as described in any one of claims 1-7, characterized in that... include: The system includes: The multimodal interaction constraint dictionary module is used to build and dynamically maintain the multimodal interaction constraint dictionary, and explicitly define the triplet structure of each exception template; The real-time feature extraction module is used to extract the multimodal text semantic set and visual perception hash code of the screen observation image at each task time step; The cross-modal interference pre-perception module is used to perform nearest neighbor search by fusing similarity metrics. When a UI anomaly is matched, the main task strategy of the large model is forcibly suspended to achieve physical blocking of the perception layer. The isolated probing and closed-loop error correction module is used to create a virtual execution environment with context isolation characteristics, perform spatial recovery actions for UI anomalies, and use the similarity of the screen structure before and after execution as a verification constraint to perform spatial coordinate penalties and iterative reflection. The Clean Observation Status Feedback Module is used to output a clean screen observation status and send it back to the main task strategy after successful status verification, so as to restore the original task flow.

9. The graphical user interface interaction error correction method based on asynchronous perception and closed-loop verification according to claim 8, characterized in that: The cross-modal interference pre-sensing module configures an interception threshold. and dual-mode equilibrium coefficient This enables physical preemption interception of contaminated images before they enter the core neural network of the agent, thereby completely eliminating contextual attention pollution caused by abnormal interference.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

Images