Interface interaction task processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an interface interaction task processing method and device, electronic equipment and storage medium, relates to the technical field of artificial intelligence, and particularly relates to the technical field of computer vision, graphical user interface automation, large model or multi-modal large model. The specific implementation scheme is: based on the interface state information of the graphical user interface, the task description information and the historical operation action, the operation intention information of the current operation action is determined; the element description information and the position of each interaction element in the graphical user interface are obtained; the operation intention information is matched with the element description information to determine the target interaction element to which the current operation action is directed from the interaction elements; and the operation instruction of the current operation action is generated based on the operation intention information and the position of the target interaction element. Based on the present scheme, the accuracy of the generated operation instruction can be ensured, and the execution success rate of the interaction task can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of computer vision, graphical user interface automation, and large model or multimodal large model technology. Specifically, this disclosure relates to a method, apparatus, electronic device and storage medium for interface interaction task processing. Background Technology

[0002] In the field of graphical user interface (GUI) automation, a graphical user interface agent can autonomously understand the current interface state based on natural language instructions, make decisions and execute specific operations to complete specific interactive tasks.

[0003] In the reasoning process of graphical user interface interaction tasks based on large models, interface images (such as screenshots) can be input into the large model, which then makes task decisions based on its visual understanding of the interface images. However, based on this visual understanding method, the large model may not be able to accurately locate the interactive elements in the graphical user interface, causing deviations in the operation and thus affecting the success rate of executing the interaction task. Summary of the Invention

[0004] To address at least one of the aforementioned deficiencies, this disclosure provides a method, apparatus, electronic device, and storage medium for processing user interface interaction tasks.

[0005] According to a first aspect of this disclosure, a method for processing interface interaction tasks is provided, the method comprising: Based on the interface state information, task description information, and historical operation actions of the graphical user interface, determine the operation intent information of the current operation action; Obtain the element description information and position of each interactive element in the graphical user interface; Match the operation intent information with the element description information to determine the target interactive element of the current operation action from the interactive elements; Based on the operation intent information and the position of the target interactive element, the operation command for the current operation action is generated.

[0006] According to a second aspect of this disclosure, a user interface interaction task processing apparatus is provided, the apparatus comprising: The operation intent determination module is used to determine the operation intent information of the current operation based on the interface state information, task description information, and historical operation actions of the graphical user interface. The interactive element information acquisition module is used to acquire the element description information and position of each interactive element in the graphical user interface; The target interactive element localization module is used to match operation intent information with element description information in order to determine the target interactive element targeted by the current operation action from the interactive elements. The operation instruction generation module is used to generate operation instructions for the current operation action based on the operation intent information and the position of the target interactive element.

[0007] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to at least one of the aforementioned processors; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable at least one processor to perform the interface interaction task processing method.

[0008] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described interface interaction task processing method.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described interface interaction task processing method.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure.

[0012] Figure 1 This is a flowchart illustrating a user interface interaction task processing method provided in an embodiment of this disclosure.

[0013] Figure 2 This is a flowchart illustrating a specific implementation of the interface interaction task processing method provided in this disclosure.

[0014] Figure 3 This is a schematic diagram of the structure of a user interface interaction task processing device provided in an embodiment of this disclosure.

[0015] Figure 4 This is a block diagram of an electronic device used to implement the interface interaction task processing method provided in the embodiments of this disclosure. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] In the reasoning process of graphical user interface interaction tasks based on a large model, in order to avoid relying on the structured hierarchical description of the graphical user interface, such as the Document Object Model (DOM) tree or the auxiliary function tree, the interface image (such as a complete screenshot of the interface) can be used as the main input of the large model. The large model completes the task decision based on the visual understanding of the interface image.

[0018] Based on the above-mentioned visual understanding method, large models may not be able to accurately locate interactive elements in the graphical user interface, especially when the element structure in the graphical user interface is complex or there are tiny interactive elements. Pure visual understanding cannot accurately anchor interactive elements, causing the generated operation to deviate from its expected position, resulting in the operation failing or producing incorrect operation results, thereby affecting the success rate of executing interactive tasks.

[0019] The interface interaction task processing method, apparatus, electronic device, and storage medium provided in the embodiments of this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0020] Figure 1 This is a flowchart illustrating the interface interaction task processing method provided in the embodiments of this disclosure, as shown below. Figure 1 As shown, the method may include the following steps: Step S110: Based on the interface state information, task description information and historical operation actions of the graphical user interface, determine the operation intent information of the current operation action; Step S120: Obtain the element description information and position of each interactive element in the graphical user interface; Step S130: Match the operation intent information with the element description information to determine the target interactive element for the current operation action from the interactive elements; Step S140: Based on the operation intent information and the position of the target interactive element, generate the operation instruction for the current operation action.

[0021] As can be seen from the above process, this disclosure directly outputs operation intent information when making operation decisions, while providing element description information and location of each interactive element. By matching the operation intent information with the element description information, the target interactive element of the current operation is determined. Then, based on the accurate location of the target interactive element and the operation intent information, an operation command is generated. This solution can improve the accuracy of locating the interactive element pointed to by the operation action, further improve the accuracy of the generated operation command, and thus improve the success rate of interactive task execution.

[0022] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments.

[0023] First, the above step S110, namely "determining the operation intent information of the current operation based on the interface state information, task description information and historical operation actions of the graphical user interface", will be described in detail with reference to the embodiments.

[0024] The interface state information refers to the data used to characterize the visual content of the current graphical user interface, which may include, but is not limited to, the interface image of the graphical user interface or visual information obtained by specific processing based on the interface image.

[0025] Task description information describes the user's final operational intent and drives the automated execution of the entire interactive task. For example, "Log in to your email and send document A to user B".

[0026] Historical actions record previously executed actions, providing historical context for the current action.

[0027] Operation intent information is used to represent the macro-level operation intent of the current operation action.

[0028] For example, the operation intent information can be a text containing a natural language description of the operation type and the target interactive element, such as "Click the search icon in the interface".

[0029] In this embodiment of the disclosure, when making decisions on operation actions based on the interface state information, task description information, and historical operation actions of the graphical user interface, the position of the interactive element to be triggered is not directly predicted. Instead, the operation intention information is output. This transforms the position prediction task into a text generation task that is easier for the model to process, thereby avoiding the problem of predicted position deviation caused by the lack of accurate positioning ability of specific interactive elements in the graphical user interface.

[0030] The following describes step S120, namely "obtaining the element description information and position of each interactive element in the graphical user interface", in detail with reference to the embodiments.

[0031] The element description information is a textual description of the function and attributes of the interactive element. Examples include "search button" and "input box".

[0032] For example, element description information and positions can be obtained by visually analyzing the interface image of a graphical user interface. For instance, interactive elements can be detected first, identifying all interactive elements such as buttons, input boxes, icons, and sliders, and determining the position of each interactive element on the interface. Position information is typically represented in the form of bounding box coordinates; for example, recording the coordinates of the top-left and bottom-right corners of the element.

[0033] In this solution, element description information and location can form a structured visual analysis set. Each data item in this set can contain the coordinates of an interactive element and its corresponding functional description. Based on this visual analysis set, high-precision positioning and accurate functional semantics of interactive elements in the graphical user interface (especially small interactive elements such as small icons or buttons) can be obtained.

[0034] The following describes in detail step S130, namely, "matching the operation intent information with the element description information to determine the target interactive element of the current operation action from the interactive elements", with reference to the embodiments.

[0035] In this step, after obtaining the abstract operation intention information and the description information and position of all interactive elements on the graphical user interface in the aforementioned steps, this step can match the macro operation intention with its corresponding interactive element on the graphical user interface, i.e., the target interactive element.

[0036] This step establishes a deep connection between natural language intent and the functionality of interface visual elements, providing a foundation for generating operation instructions for the matched target interactive elements.

[0037] The following describes step S140, namely "generating an operation instruction for the current operation action based on the operation intent information and the position of the target interactive element", in detail with reference to the embodiments.

[0038] Once the target interactive element is determined, its precise pixel position can be obtained. Then, an executable atomic operation instruction can be generated based on the operation intent information and the position of the target interactive element.

[0039] Specifically, the operation type (e.g., click, swipe, long press, text input) extracted from the operation intent information can be assembled with the specific coordinates of the target interactive element. For example, if the operation intent is "click" and the center point of the bounding box of the target interactive element is (x, y), then the generated final operation instruction can correspond to the operation of "perform a click operation at coordinates (x, y)".

[0040] In summary, based on steps S110 to S140, this disclosure directly outputs operation intent information during operation action decision-making, while providing element description information and location of each interactive element. By matching the operation intent information with the element description information, the target interactive element for the current operation action is determined, thereby obtaining the accurate location information of the target interactive element. Finally, operation instructions can be generated based on the accurate location of the target interactive element and the operation intent information. Compared to the method in related technologies that directly predicts the coordinates of the target interactive element based on visual understanding of the image user interface, this solution provides the functional description and accurate location of the interactive element, enabling the operation intent to be accurately matched to the target interactive element, thereby obtaining the accurate coordinates of the target interactive element. This improves the accuracy of the target interactive element's location, further enhancing the accuracy of the generated operation instructions and thus increasing the success rate of the interactive task execution.

[0041] In one optional embodiment of this disclosure, the operation intent information is matched with the element description information to determine the target interactive element for the current operation action from the interactive elements, including: The target interactive element is determined based on the semantic similarity between the operation intention information and the description information of each element.

[0042] In this embodiment of the disclosure, the target interactive element that best matches the operation intent information semantically can be searched based on the semantic similarity between element description information and operation intent information. For example, when the operation intent information is "click the search icon in the upper right corner of the page", this operation intent information can be compared with all element description information. By calculating the semantic similarity between them, elements described as "search icon" or having similar functions can be identified.

[0043] In this solution, the semantic similarity-based matching method can accurately understand the deep relationship between natural language intent and the function of interface visual elements. It can avoid situations where simple keyword or literal matching cannot handle the differences in expression between natural language description and interface element function text, such as the operation intent information "search" not matching the element description information "magnifying glass". This effectively improves the accuracy of target element positioning.

[0044] For example, a pre-trained semantic model can be used to map the text of the operational intent and the text of the element descriptions of each interactive element into a high-dimensional semantic vector space, and then the spatial distance (e.g., cosine similarity) between these vectors can be calculated. The closer the distance, the more semantically similar the two are.

[0045] When matching based on semantic similarity, the presence of multiple semantically similar interactive elements in a graphical user interface (e.g., multiple "settings" buttons) may prevent the unique identification of the target interactive element. In one optional approach of this disclosure, the operation intent information further includes location description information of the target interactive element. The target interactive element is determined based on the semantic similarity between the operation intent information and the description information of each element, including: The target interactive element is determined based on the semantic similarity between the operation intent information and the description information of each element, and based on the location description information.

[0046] Location description information refers to the description of the target interactive element's position in the graphical user interface. Location description information can be part of the operation intent information. For example, if the operation intent information is "Click the search icon in the upper right corner of the page", the location description information would be "upper right corner of the page".

[0047] In this step, location description information is introduced as a constraint based on semantic comparison, which enables accurate location of the target element when there are multiple semantically similar candidate interactive elements.

[0048] For example, the semantic similarity between the operation intent information and the element description information can be calculated first. Then, interactive elements corresponding to element description information whose semantic similarity meets a preset condition (e.g., greater than a set value) are identified as candidate interactive elements. Next, it is determined whether each candidate interactive element satisfies the aforementioned location description information, for example, whether it is located in the upper right corner of the graphical user interface. After determining that a candidate interactive element satisfies the aforementioned location description information, it can be identified as the target interactive element.

[0049] In this solution, by combining the above-mentioned matching method with location description information, it is possible to effectively eliminate interference elements in the graphical user interface that are semantically similar but do not match in location, and achieve accurate disambiguation of multiple candidate elements, thereby ensuring that the target interactive element can be accurately identified in a complex graphical user interface.

[0050] In one alternative embodiment of this disclosure, the location description information includes a relative location description determined based on a graphical user interface or a reference object within it.

[0051] The relative position description can be used to describe the orientation of an interactive element relative to the entire graphical user interface, such as "top right corner of the interface" or "bottom center of the screen". In addition, the relative position description can also be defined based on a specific reference object within the graphical user interface, such as "third item in the search results list" or "below the search button in the sidebar".

[0052] For example, the above-mentioned operation intent information can be based on the output of the large model, and the prompts of the large model can instruct the large model to generate operation intent information containing a description of the relative position.

[0053] In this embodiment, the location description information does not use absolute pixel coordinates, but rather a relative location description. This method allows large models to output location descriptions of interactive elements in a way that more closely resembles human expression habits, thereby reducing the model's reliance on high-precision location data. Simultaneously, during the target interactive element selection process, the system can quickly locate content that meets the conditions based on this semantic location constraint, improving matching efficiency and accuracy.

[0054] In one optional embodiment of this disclosure, the operation intent information of the current operation is determined based on the interface state information of the graphical user interface, task description information, and historical operation actions, including: The multimodal large model is invoked to determine the operation intent information of the current operation based on interface state information, task description information, historical operation actions and element description information, or based on interface state information, task description information, historical operation actions, element description information and the position of interactive elements.

[0055] In particular, when making action decisions using a multimodal large model, richer contextual inputs can be provided. This contextual information can include not only interface state information, task description information, and historical actions, but also element description information, or both element description information and the position of interactive elements.

[0056] Element descriptions and the location of interactive elements accurately represent their function and position. Explicitly injecting these into the input of a multimodal large model helps the model understand the current interface layout and available functions, thus facilitating the generation of more specific and precise operational intentions. For example, based on the relatively general operational intention information of "click the submit button," the output could be more precise as "click the submit button in the lower right corner." Here, "lower right corner" serves as a location description, working in conjunction with the semantic description "submit," enabling the system to uniquely identify the target among multiple submit buttons, resolving the ambiguity problem of pure semantic matching.

[0057] In this solution, by adding element description information and the position of interactive elements as auxiliary criteria, the accuracy and specificity of operation intent information can be effectively improved, which helps to perform high-precision matching based on operation intent information in the future.

[0058] In one optional approach of this disclosure, obtaining the element description information and position of each interactive element in the graphical user interface includes: Detect interactive elements from the interface image of the graphical user interface and determine the position of the interactive elements; Extract the image of each interactive element based on its location. Visual language understanding is performed on the images of interactive elements to generate element description information for each interactive element.

[0059] The process begins by detecting interactive elements in the interface image. Then, each interactive element is precisely located and extracted from the image, forming the interactive element image. Extracting independent images of each interactive element ensures that subsequent visual language understanding steps can focus on the visual details of individual interactive elements, unaffected by complex backgrounds.

[0060] In the visual language understanding step, interactive elements can be visually understood in a way that simulates human eye observation, and then natural language text can be generated.

[0061] For example, a lightweight interactive region detection network can be used to detect interactive elements. This network can be an object detection model such as You Only Look Once (YOLO) or DetectionTransformer (DETR). The network can be specifically trained to recognize various interactive elements in a graphical user interface, such as buttons, text boxes, sliders, and icons. The network can perform detection based on the uncompressed raw interface image, outputting a list of bounding box coordinates for all detected interactive elements. Then, based on each bounding box, a corresponding local image patch, i.e., the interactive element image, is cropped from the raw image. This interactive element image can be fed into a lightweight visual language understanding model, which typically includes a visual encoder and a text decoder, capable of converting the input interactive element image into phrases describing its function, such as "username input box," "password input box," and "login button."

[0062] Based on this solution, high-quality element description information can be obtained accurately and reliably, providing a reliable data foundation for subsequent operational decisions.

[0063] To ensure the safety and effectiveness of the current operation, in one optional embodiment of this disclosure, after generating the operation instruction for the current operation, the method further includes: The current operation is validated based on whether there are valid interactive elements in the interactive area targeted by the current operation, and / or whether the current operation belongs to a preset risk operation.

[0064] Specifically, the validity of the current operation is ensured by checking whether there are valid interactive elements in the interactive area targeted by the current operation, i.e., whether there are operable interactive elements at the operation location.

[0065] Verifying whether the current operation is a preset risky operation can ensure the safety of the current operation.

[0066] For example, risky operations may include, but are not limited to, formatting a disk, deleting system folders, and clearing a database.

[0067] When the aforementioned verification process detects that the current operation fails verification, in addition to blocking the execution of the operation, the system also needs to learn from the error and generate a more reasonable alternative action. To address this issue, in one optional embodiment of this disclosure, after verifying the current operation, the method further includes: An error feedback message is generated in response to the current operation failing the validation. Based on the interface state information, task description information, and historical operation actions of the graphical user interface, the operation intent information of the current operation action is determined, including: Based on interface status information, task description information, historical operation actions, and error feedback information, determine the operation intent information of the current operation action.

[0068] The error feedback information describes the details of the failed operation, such as "Invalid action, no button control was detected at the target coordinates" or "The action was blocked, this operation is a high-risk behavior".

[0069] Error feedback information can be associated with historical operation records and used together as re-input to the multimodal large model as historical context, thereby prompting the multimodal large model to avoid operations that have been proven to be wrong and to plan a new, more reliable operation.

[0070] In this solution, after the current operation fails to be verified, error feedback information is generated and provided to the multimodal large model so that the multimodal large model can reflect and self-correct, thereby effectively preventing the interactive task from being terminated due to the failure of a single operation and improving the success rate of the interactive task.

[0071] In one optional embodiment of this disclosure, the interface state information includes a visual representation of the interface, and the above method further includes: Extract the local background image corresponding to the interface background from the interface image of the graphical user interface; Feature compression processing is performed on the local background image; A visual representation of the interface is constructed based on the local background image after feature compression and the parts of the interface image other than the local background image.

[0072] The interface background refers to a continuous area in the interface image that does not contain interactive elements. This can be a large area filled with a solid color or gradient, or it can encompass areas with repetitive, regular textures but lacking interactive semantics (such as watermark backgrounds or decorative patterns). Interface backgrounds typically have highly consistent visual features, contain relatively little effective information, and contribute little to understanding interactive tasks.

[0073] It is understood that the specific visual style of the interface background is not limited in the embodiments disclosed herein.

[0074] In graphical user interfaces, the interface image often contains both interactive elements that require focused attention (such as buttons, input boxes, icons, etc.) and background areas that do not require focused attention. In order to reduce redundant information in the subsequent visual understanding process, it is first necessary to separate the local background images of these background areas from the interface image.

[0075] After acquiring the local background image, feature compression is required to reduce the amount of visual representation data input to subsequent inference processes. Since the local background image itself has low information density, moderate compression of its features will not significantly affect the overall understanding of the interface image.

[0076] Feature compression refers to processing techniques that reduce the amount of data related to the visual features of a local background image. For example, the features of a local background image can be transformed to a more compact feature dimension.

[0077] Interface visual representation is the overall representation of a graphical user interface, serving as the foundational information for visual understanding in subsequent reasoning processes.

[0078] Specifically, the local background image after feature compression can be fused and recombined with the image of the non-interface background area that retains the original details to obtain the visual representation of the interface.

[0079] For example, the entire interface image can be divided into multiple image blocks according to a preset size, and each image block serves as a visual token. The multiple visual tokens corresponding to the feature-compressed local background image cover the original image block area, and then are concatenated with the visual tokens of the non-background area (i.e., the area with dense interactive elements) to form an ordered sequence of visual tokens, i.e., the visual representation of the interface.

[0080] In this step, by fusing the compressed local background image with the image of the non-interface background area, the visual features of the interface can contain both the overall layout and contextual summary information provided by the compressed background and the clear local visual features of key interactive elements. This achieves a balance between global context and local details, ensuring a good visual understanding effect.

[0081] Compared with the method in related technologies that inputs the entire interface image into the large model without distinction, the interface visual representation constructed in this embodiment of the present disclosure has a significantly shorter total length of the token sequence due to the compression of the background part, thus effectively reducing the computational overhead and inference latency of the multimodal large model.

[0082] In this solution, the interface visual representation is constructed by compressing the features of the interface background, which significantly reduces the amount of visual feature data while still being able to accurately understand the current interface state and make accurate operational decisions. This effectively solves the problem of excessive computational resource consumption and increased response latency caused by directly using interface images for visual understanding.

[0083] In one optional embodiment of this disclosure, extracting a local background image corresponding to the interface background from the interface image of the graphical user interface includes: Divide the interface image into multiple image blocks with preset sizes; Local background images are determined based on image blocks that are adjacent in location and have a content similarity greater than a preset first similarity threshold.

[0084] Interface backgrounds typically appear as large, continuous image regions with simple visual features, while foreground interactive elements usually appear as smaller image regions with drastic feature variations and discrete distribution. When detecting interface backgrounds, the interface image can be finely segmented into regular, uniformly sized processing units (image blocks). Then, analysis is performed based on the visual similarity and connectivity of these image blocks to accurately identify the interface background from the interface image.

[0085] For example, the original interface image can be divided into a grid to obtain multiple image blocks of preset sizes. The preset size can be set according to the resolution of the input image and the desired processing granularity. For example, for a 1920×1080 pixel screen image, the image block size can be set to 32×32 pixels or 64×64 pixels. Then, for each image block, a visual feature extraction network is used to extract visual feature vectors. Next, all image blocks are traversed, and for any pair of spatially adjacent (vertical or horizontal) image blocks, the similarity between their visual feature vectors is calculated, for example, cosine similarity can be calculated. If the similarity exceeds a preset first similarity threshold (e.g., 0.95), the two image blocks are considered to have highly consistent content and may belong to the same continuous background. The continuous area covered by adjacent image blocks that meet the above conditions is identified as a local background image.

[0086] In this scheme, by adopting an image segmentation and similarity merging approach, the interface background can be adaptively and accurately identified, providing high-quality input for subsequent accurate feature compression.

[0087] When identifying interface backgrounds based on image patch content similarity, some interactive elements may have high internal similarity. For example, in a neatly arranged table, the individual cells may have visually similar textures or colors. These interactive element areas may be incorrectly identified as interface backgrounds and subjected to feature compression, leading to the loss of important interactive information. To address this misjudgment problem, one optional method of this disclosure involves determining a local background image based on image patches that are geographically adjacent and have a content similarity greater than a preset first similarity threshold, including: Image blocks that are adjacent in location and have a content similarity greater than a preset first similarity threshold are combined into candidate local images; Determine the content complexity of the candidate local image; Candidate local images with content complexity below a preset content complexity threshold are identified as local background images.

[0088] Content complexity is used to represent the richness of information contained within a candidate local image. Various quantification metrics can be used for content complexity, such as the variance of pixel grayscale values ​​or information entropy. The complexity threshold can be determined empirically or through calibration on a dataset.

[0089] The content within the actual interface background is consistent and simple, with extremely low metrics such as information entropy and gradient magnitude. Interactive elements that might be misidentified based solely on visual similarity may have similar internal content, but still contain rich visual information. For example, the cells or cell boundaries in a neatly arranged table contain rich, complex information such as edges and text. By calculating the content complexity of the entire candidate local image and excluding candidate local images with higher content complexity, the accuracy of the identified interface background can be effectively improved.

[0090] For example, through similarity-based image patch merging, a table region spanning multiple rows and columns is identified as a large candidate local image. To determine whether it is indeed an interface background, the average gradient magnitude of all pixels within the candidate local image can be calculated as content complexity. For a table containing text and borders, there are abundant vertical and horizontal edges, so its average gradient magnitude will be relatively high. If a content complexity threshold is predefined (e.g., an average gradient magnitude below 15 can be considered a solid color background), then the gradient magnitude of the table candidate region will be much higher than this threshold, thus being excluded and not identified as a local background image. Conversely, when the candidate local image is a continuous solid color blank area, its average gradient magnitude is close to zero and will be identified as a local background image.

[0091] In this solution, by adding a filtering mechanism based on content complexity, it is possible to ensure that only smooth backgrounds with no information are compressed, thereby avoiding the misidentification of visually similar but information-dense interactive elements and effectively improving the recognition accuracy of interface backgrounds.

[0092] When merging image blocks based on similarity in an interface background region, using a single, high similarity threshold may lead to image blocks belonging to the same continuous background not being merged due to minor differences caused by image compression artifacts, subtle gradient transitions, etc., resulting in fragmentation or omission of the background region and affecting the accuracy of interface background recognition. To solve this problem, one optional method disclosed herein determines a local background image based on image blocks that are adjacent in location and have a content similarity greater than a preset first similarity threshold, including: Image blocks that are adjacent in location and whose content similarity meets a preset first similarity threshold are combined into an initial local image; Image blocks that are adjacent to the initial local image and whose content similarity with the initial local image is greater than the second similarity threshold are merged into the initial local image to obtain a local background image, wherein the second similarity threshold is less than the first similarity threshold.

[0093] In this embodiment, a pair of similarity threshold mechanisms with different levels of similarity are introduced. First, a stricter high threshold (the first similarity threshold) is used to identify regions with highly consistent content; this region is actually the core area of ​​the interface background. Then, a more lenient low threshold (the second similarity threshold) is used to filter edge image patches adjacent to this core region. These edge image patches may deviate slightly from the core region due to lighting, rendering, or other reasons, but should still be considered part of the overall background visually.

[0094] For example, based on a first similarity threshold (e.g., cosine similarity > 0.98), a solid-color background region consisting of a group of adjacent image patches is selected, which is the core region of the interface background. There is a slight shadow transition area around this core region. The image patches in this transition area do not meet the first similarity threshold, but they do meet a second similarity threshold (e.g., cosine similarity > 0.85). In this case, the transition area can be included in the core region to collectively form a larger and more complete interface background.

[0095] In this solution, based on the above mechanism, the interface background can be progressively derived from the core to the edge, thereby avoiding the omission of background areas due to subtle differences in visual features, and thus obtaining a more complete and continuous local background image.

[0096] Different graphical user interfaces (GUIs) exhibit vastly different levels of content complexity. For example, a simple desktop interface might have over 90% solid color backgrounds, while a complex product details page could be filled with a large amount of data and images. Using a fixed similarity threshold for background filtering is ineffective in handling scenarios with significantly varying content complexity. To address this issue, in one optional approach disclosed herein, both the first and second similarity thresholds are negatively correlated with the content complexity of the interface image.

[0097] The content complexity of the interface image can be calculated by performing content complexity calculations on the entire interface image or its sampled regions. For example, the average information entropy of all pixels in the entire interface image can be calculated.

[0098] This solution provides a mechanism for dynamically adjusting the first similarity threshold and the second similarity threshold based on the content complexity of the interface image. This mechanism can adapt to interfaces with vastly different content complexities and improve the accuracy of interface background recognition.

[0099] Specifically, for interfaces with low content complexity, such as those with large areas of solid color and only a few icons, the visual information they contain is limited, and the background content is highly homogeneous. In this case, a higher similarity threshold can be used to merge only those image blocks with high similarity into local background images. Conversely, for interfaces with high content complexity, such as those full of various controls and textures, in order to ensure the identification of the interface background (which may be small blank spaces between multiple interactive elements), the similarity threshold can be lowered to relax the merging conditions, ensuring that as many of these fragmented and discrete background areas as possible can be identified.

[0100] For example, content complexity may have a linear negative correlation with the first similarity threshold or the second similarity threshold, that is, the higher the content complexity, the lower the first similarity threshold or the second similarity threshold; the lower the content complexity, the higher the first similarity threshold or the second similarity threshold.

[0101] When performing feature compression on identified local background images, applying uniform pooling indiscriminately to the entire background region may result in the smooth loss of certain key information. For example, background regions near interactive elements often contain subtle projections, highlighted edges, or dividing lines. While this information does not constitute the interactive element itself, it plays a crucial role in helping large models understand the hierarchical structure and spatial relationships of interactive elements. To address the issue of information loss that may result from uniform pooling, one optional approach disclosed herein involves performing feature compression processing on the local background image, including: The pooling weights are determined based on the position and / or content complexity of each image patch in the local background image. Based on pooling weights, weighted pooling is performed on each image block in the local background image.

[0102] The position and / or content complexity of each image block in the local background image can reflect the importance of each image block in the local background image. Higher pooling weights can be assigned to image blocks with higher importance (such as those close to interactive elements or those with relatively complex internal textures), while lower pooling weights can be assigned to image blocks with lower importance (such as those located in the middle of the background or those with overly simple textures), so that the visual information of image blocks with higher importance can be better preserved.

[0103] For example, the pooling weights can be negatively correlated with the distance of an image patch to the nearest interactive element. For instance, when the distance to the nearest interactive element is less than a preset pixel value (e.g., 20 pixels), the pooling weight can be set to 1.0, while the pooling weight of an image patch that is more than the preset pixel value can decrease logarithmically with the distance.

[0104] For example, the size of the pooling weights can be positively correlated with the content complexity of the image patch itself.

[0105] Weighted pooling refers to pooling each image block based on its corresponding pooling weight to obtain a compressed visual token.

[0106] In this scheme, the importance of image patches is accurately measured based on their location and content complexity, and the pooling weights are dynamically adjusted based on the importance. This allows important information in the background to be preserved as much as possible during feature compression, which helps improve the accuracy of visual understanding of the interface.

[0107] For example, Figure 2 This is a flowchart illustrating a specific implementation of the interface interaction task processing method provided in this disclosure.

[0108] like Figure 2 As shown, the first step is to obtain a screenshot of the graphical user interface at the current moment as the interface image. Then, the background image of a local area is extracted from the interface image.

[0109] Specifically, the interface image can be evenly divided into multiple image blocks with preset sizes.

[0110] Based on the visual feature similarity between image blocks, image blocks that are adjacent in position and have a similarity higher than a first preset threshold are merged to form candidate background regions. The content complexity of the candidate background regions is evaluated, and only simple regions with content complexity lower than the preset threshold and sparse information are determined as the final background regions, i.e., local background images.

[0111] Alternatively, a dual similarity thresholding mechanism can be employed. After identifying the core region of the interface background using a higher first similarity threshold, image patches adjacent to the core region are then identified as transition regions and included in the local background image based on a lower second similarity threshold, in order to obtain a more complete and continuous local background image.

[0112] After identifying the local background image, feature compression can be performed on the local background image, such as weighted pooling of image blocks in the local background image.

[0113] Optionally, background blocks closer to interactive elements or with slightly higher content complexity are given higher pooling weights to retain more details and spatial location cues while compressing.

[0114] The compressed background features are fused with the uncompressed interactive element features (excluding the background) in the interface image to construct a complete visual token sequence, i.e., the interface visual representation, which serves as the input for subsequent decision-making modules.

[0115] Furthermore, interactive elements can be parsed based on the interface images.

[0116] Specifically, the detection network can identify and locate interactive elements (such as buttons, input boxes, icons, etc.) in the interface and output their bounding box coordinates. Based on the bounding box coordinates, the interactive element image of each interactive element is extracted from the interface image. A visual language model (such as a lightweight caption generation unit) is used to perform visual semantic understanding on each interactive element image, generating element description information (such as "login button" and "search icon") to describe its functional attributes, and recording its corresponding position coordinates.

[0117] Then, the multimodal large model decision-making action can be invoked.

[0118] Specifically, the aforementioned interface visual representation, element description information, and location information, along with the task description and action sequences of historical operations, can be input into a multimodal large model. Based on this, the model outputs the operation intent information of the current operation.

[0119] After obtaining the operation intent information, matching logic can be executed to determine the target interactive element. Specifically, semantic similarity can be calculated between the operation type in the intent description information and the description information of all elements. Simultaneously, positional matching degree can be calculated between the positional description information in the intent description information and the actual bounding box coordinates of each interactive element. Then, a comprehensive decision is made by weighted and fused the semantic similarity and positional matching degree to obtain a comprehensive score. The element with the highest comprehensive score is selected as the target interactive element. For example, the system will accurately select the "search button" located in the upper right corner of the screen, rather than a search button located elsewhere on the interface.

[0120] After matching the target interactive element, the final, executable atomic operation instruction can be generated based on the position of the target interactive element (such as coordinates) and the operation type in the operation intent (such as "click").

[0121] After generating the operation command for the current operation, the current operation can be verified.

[0122] Specifically, the current operation can undergo dual verification for both validity and security. The validity verification can determine whether a valid interactive element exists at the target location pointed to by the operation. The security verification can determine whether the operation belongs to a preset high-risk operation behavior (such as "delete" or "format").

[0123] If the verification passes, the current operation can be performed.

[0124] If the validation fails, structured error feedback information can be generated, such as "No valid interactive element at the target location" or "High-risk operation detected." The error feedback information is appended to the historical sequence of executed actions, driving the multimodal large model to re-make action decisions based on the updated historical context (including error feedback information).

[0125] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0126] According to another embodiment, a user interface interaction task processing device is provided. Figure 3 A schematic diagram of the interface interaction task processing device according to one embodiment is shown. Figure 3 As shown, the interface interaction task processing device 300 includes: The operation intent determination module 310 is used to determine the operation intent information of the current operation based on the interface state information, task description information and historical operation actions of the graphical user interface. The interactive element information acquisition module 320 is used to acquire the element description information and position of each interactive element in the graphical user interface; The target interactive element localization module 330 is used to match the operation intent information with the element description information in order to determine the target interactive element targeted by the current operation action from the interactive elements. The operation instruction generation module 340 is used to generate operation instructions for the current operation action based on the operation intent information and the position of the target interactive element.

[0127] As an optional approach, the target interactive element positioning module 330 is specifically used for: The target interactive element is determined based on the semantic similarity between the operation intention information and the description information of each element.

[0128] As an optional approach, the operation intent information also includes location description information of the target interactive element. When determining the target interactive element based on the semantic similarity between the operation intent information and the description information of each element, the target interactive element localization module 330 is specifically used for: The target interactive element is determined based on the semantic similarity between the operation intent information and the description information of each element, and based on the location description information.

[0129] As an alternative, location description information includes a relative location description determined based on a graphical user interface or a reference object within it.

[0130] As an optional approach, the operation intent determination module 310 is specifically used for: The multimodal large model is invoked to determine the operation intent information of the current operation based on interface state information, task description information, historical operation actions and element description information, or based on interface state information, task description information, historical operation actions, element description information and the position of interactive elements.

[0131] As an optional method, the interactive element information acquisition module 320 is specifically used for: Detect interactive elements from the interface image of the graphical user interface and determine the position of the interactive elements; Extract the image of each interactive element based on its location. Visual language understanding is performed on the images of interactive elements to generate element description information for each interactive element.

[0132] As an optional approach, the above-mentioned device also includes an action verification module (not shown in the figure), used for: After generating the operation instruction for the current operation, the current operation is verified based on whether there are valid interactive elements in the interactive area targeted by the current operation, and / or whether the current operation belongs to a preset risk operation.

[0133] As an optional embodiment, the above-mentioned device also includes an action replanning module (not shown in the figure), used for: After validating the current operation, an error feedback message is generated in response to the current operation failing the validation. The operation intent determination module 310 is specifically used for: Based on interface status information, task description information, historical operation actions, and error feedback information, determine the operation intent information of the current operation action.

[0134] As an optional approach, the interface state information includes a visual representation of the interface, and the aforementioned device further includes a feature compression module (not shown in the figure), used for: Extract the local background image corresponding to the interface background from the interface image of the graphical user interface; Feature compression processing is performed on the local background image; A visual representation of the interface is constructed based on the local background image after feature compression and the parts of the interface image other than the local background image.

[0135] As an optional approach, when the operation intent determination module 310 extracts the local background image corresponding to the interface background from the interface image of the graphical user interface, it is specifically used for: Divide the interface image into multiple image blocks with preset sizes; Local background images are determined based on image blocks that are adjacent in location and have a content similarity greater than a preset first similarity threshold.

[0136] As an optional approach, the feature compression module (not shown in the figure) is specifically used to determine local background images based on image blocks that are geographically adjacent and have a content similarity greater than a preset first similarity threshold, for the following purposes: Image blocks that are adjacent in location and have a content similarity greater than a preset first similarity threshold are combined into candidate local images; Determine the content complexity of the candidate local image; Candidate local images with content complexity below a preset content complexity threshold are identified as local background images.

[0137] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0138] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0139] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0140] Figure 4A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0141] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0142] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0143] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the interface interaction task processing method described above. For example, in some embodiments, the above-described interface interaction task processing method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the interface interaction task processing method described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the interface interaction task processing method described above by any other suitable means (e.g., by means of firmware).

[0144] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0146] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0148] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0149] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0150] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the above-described interface interaction task processing method.

[0151] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described interface interaction task processing method.

[0152] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0153] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for handling user interface interaction tasks, comprising: Based on the interface state information, task description information, and historical operation actions of the graphical user interface, determine the operation intent information of the current operation action; Obtain the element description information and position of each interactive element in the graphical user interface; The operation intent information is matched with the element description information to determine the target interactive element for the current operation action from the interactive elements; Based on the operation intent information and the position of the target interactive element, an operation instruction for the current operation action is generated.

2. The method according to claim 1, wherein, The step of matching the operation intent information with the element description information to determine the target interactive element for the current operation action from the interactive elements includes: The target interactive element is determined based on the semantic similarity between the operation intent information and the description information of each element.

3. The method according to claim 2, wherein, The operation intent information also includes the location description information of the target interactive element. Determining the target interactive element based on the semantic similarity between the operation intent information and each element description information includes: The target interactive element is determined based on the semantic similarity between the operation intent information and the description information of each element, and based on the location description information.

4. The method according to claim 3, wherein, The location description information includes a relative location description determined based on the graphical user interface or a reference object within it.

5. The method according to claim 3 or 4, wherein, The operation intent information for determining the current operation based on the interface state information, task description information, and historical operation actions of the graphical user interface includes: The multimodal large model is invoked to determine the operation intent information of the current operation based on the interface state information, task description information, historical operation actions, and element description information, or based on the interface state information, task description information, historical operation actions, element description information, and the position of the interactive element.

6. The method according to any one of claims 1-5, wherein, The step of obtaining the element description information and position of each interactive element in the graphical user interface includes: The interactive elements are detected from the interface image of the graphical user interface and the position of the interactive elements is determined. Extract the image of each interactive element based on its position. Visual language understanding is performed on the images of the interactive elements to generate element description information for each interactive element.

7. The method according to any one of claims 1-6, wherein, After generating the operation instruction for the current operation action, the method further includes: The current operation is verified based on whether there are valid interactive elements in the interactive area targeted by the current operation, and / or whether the current operation belongs to a preset risk operation.

8. The method according to claim 7, further comprising, after verifying the current operation, the method including: In response to the current operation failing the verification, an error feedback message is generated; The operation intent information for determining the current operation based on the interface state information, task description information, and historical operation actions of the graphical user interface includes: Based on the interface status information, task description information, historical operation actions, and error feedback information, the operation intent information of the current operation action is determined.

9. The method according to any one of claims 1-7, wherein the interface state information includes an interface visual representation, and the method further includes: Extract the local background image corresponding to the interface background from the interface image of the graphical user interface; The local background image is subjected to feature compression processing; The visual representation of the interface is constructed based on the local background image after feature compression and the portion of the interface image excluding the local background image.

10. The method according to claim 9, wherein, Extracting the local background image corresponding to the interface background from the interface image of the graphical user interface includes: The interface image is divided into multiple image blocks with preset sizes; The local background image is determined based on image blocks that are adjacent in location and have a content similarity greater than a preset first similarity threshold.

11. The method according to claim 10, wherein, The determination of the local background image based on image blocks that are geographically adjacent and have a content similarity greater than a preset first similarity threshold includes: Image blocks that are adjacent in location and have a content similarity greater than a preset first similarity threshold are combined into candidate local images; Determine the content complexity of the candidate local image; Candidate local images whose content complexity is lower than a preset content complexity threshold are identified as local background images.

12. A user interface interaction task processing device, comprising: The operation intent determination module is used to determine the operation intent information of the current operation based on the interface state information, task description information, and historical operation actions of the graphical user interface. An interactive element information acquisition module is used to acquire the element description information and position of each interactive element in the graphical user interface; The target interactive element localization module is used to match the operation intent information with the element description information to determine the target interactive element targeted by the current operation action from the interactive elements. The operation instruction generation module is used to generate the operation instruction for the current operation action based on the operation intent information and the position of the target interactive element.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images