Graphical user interface training data generation method, graphical user interface agent training method and device, equipment and medium

Abstract

This disclosure provides a method for generating graphical user interface (GUI) training data, a method and apparatus for training GUI intelligent agents, and a medium, relating to the field of computer technology. The method includes: determining the coordinates of effective element regions by detecting interface elements in a GUI image and combining this with region segmentation; performing local cropping based on the effective element region coordinates to obtain local sub-images; extracting fine-grained semantic labels of interface elements under multi-dimensional prompts using a visual language model, and constructing interactive operation trajectory data based on these fine-grained semantic labels; and constructing GUI training data using the GUI image, effective element region coordinates, fine-grained semantic labels, and interactive operation trajectory data. This solution can improve the reliability, completeness, and fine-grainedness of the training data, thereby enhancing the automated interactive decision-making capabilities of the GUI intelligent agent.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a method for generating graphical user interface training data, a method, apparatus, device, and medium for training graphical user interface intelligent agents. Background Technology

[0002] With the rapid development of artificial intelligence technology, graphical user interface (GUI) agents are gradually gaining attention in applications such as automated operation and assisted interaction. To enable agents to understand GUI content and perform corresponding operations, it is typically necessary to construct training data containing interface images, element location information, and semantic descriptions. The completeness and accuracy of the training data directly affect the decision-making performance of the agent model.

[0003] Currently, in the process of constructing high-quality training data for graphical user interfaces, the lack of precise perception of the local details of the graphical user interface leads to coarse semantic granularity, inaccurate coordinates of interface elements, and an inability to reflect the real interaction state between interface elements, thus affecting the training effect and decision stability of the graphical user interface agent.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method for generating training data for a graphical user interface (GUI), a method and apparatus for training a graphical user interface (GUI) intelligent agent, and a medium, thereby improving the reliability, integrity, and granularity of the training data, and thus enhancing the automated interactive decision-making capabilities of the GUI intelligent agent.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to a first aspect of the present disclosure, a method for generating graphical user interface training data is provided, comprising: Acquire the graphical user interface image to be processed, and determine the effective element region coordinates corresponding to each interface element in the graphical user interface image; The graphical user interface image is cropped locally based on the coordinates of the effective element region to obtain a local sub-image corresponding to each of the interface elements. Fine-grained semantic tags are extracted from each interface element through the local sub-image, and interactive operation trajectory data corresponding to the graphical user interface image are determined based on the fine-grained semantic tags. The graphical user interface training data is generated based on the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interactive operation trajectory data.

[0008] In some example embodiments of this disclosure, based on the foregoing scheme, determining the effective element region coordinates corresponding to each interface element in the graphical user interface image includes: The graphical user interface image is subjected to interface element detection to obtain the initial element region coordinates of each interface element; Based on the initial element region coordinates, the graphical user interface image is segmented to obtain candidate mask images for each interface element. Based on the degree of matching between the initial element region coordinates and the candidate mask image, a valid mask image is selected from the candidate mask images, and the valid element region coordinates are generated using the valid mask image.

[0009] In some example embodiments of this disclosure, based on the foregoing scheme, the step of filtering valid mask images from the candidate mask images according to the degree of matching between the initial element region coordinates and the candidate mask images includes: Calculate the degree of overlap between the initial element region coordinates and the candidate mask image; Candidate mask images with an overlap degree greater than or equal to a preset overlap degree threshold are used as valid mask images.

[0010] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the step of detecting interface elements in the graphical user interface image to obtain the initial element region coordinates of each interface element includes: The graphical user interface image is input into a trained point prediction multimodal model to obtain the initial element region coordinates of each interface element; the point prediction multimodal model is used to perform global detection on the graphical user interface image.

[0011] In some example embodiments of this disclosure, based on the foregoing scheme, the step of performing region segmentation processing on the graphical user interface image based on the initial element region coordinates to obtain candidate mask images for each interface element includes: The graphical user interface image and the initial element region coordinates are input into a trained cue-driven segmentation model to obtain candidate mask images for each interface element; the cue-driven segmentation model is used to perform element mask recognition on the graphical user interface image using the initial element region coordinates as cue targets.

[0012] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the step of cropping the graphical user interface image locally according to the coordinates of the effective element region to obtain a local sub-image corresponding to each interface element includes: The initial cropping area is determined based on the coordinates of the effective element area, and the target cropping area is obtained by expanding the area outward along the periphery of the initial cropping area by a preset width ratio. The graphical user interface image is cropped according to the target cropping region to obtain local sub-images corresponding to each interface element.

[0013] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the step of extracting fine-grained semantic tags for each interface element through the local sub-image includes: Perform multi-dimensional semantic analysis on the local sub-image to generate multi-dimensional prompt questions targeting the interface element features in the local sub-image; Based on the multi-dimensional prompt questions and the local sub-images, interface element feature answer data corresponding to the local sub-images is generated; The multi-dimensional prompt questions and the interface element feature answer data are used as fine-grained semantic labels for interface elements in the local sub-image.

[0014] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the interface element features include at least one of color features, brightness features, shape features, and interaction features; The interactive features include active state, disabled state, selected state, or highlighted state.

[0015] In some example embodiments of this disclosure, based on the foregoing scheme, determining the interaction trajectory data corresponding to the graphical user interface image based on the fine-grained semantic tags includes: Determine the target interaction result corresponding to the graphical user interface image; Based on the fine-grained semantic tags, determine the state conditions that at least some interface elements need to satisfy when achieving the target interaction result; Based on the fine-grained semantic tags, an interaction step sequence corresponding to at least some interface elements is constructed when the state conditions are met, and the interaction step sequence is used as the interaction operation trajectory data corresponding to the graphical user interface image.

[0016] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the step of constructing the sequence of interaction steps corresponding to at least some interface elements when satisfying the state conditions based on the fine-grained semantic tags includes: Based on the fine-grained semantic tags, determine the state dependencies between at least some interface elements, and determine the logical operation order corresponding to at least some of the interface elements according to the state dependencies; The interactive actions corresponding to at least some of the interface elements when the state conditions are met are determined, and the interactive actions are combined according to the logical operation sequence to form the interactive step sequence.

[0017] According to a second aspect of the present disclosure, a graphical user interface agent training method is provided, comprising: Obtain graphical user interface training data generated by the graphical user interface training data generation method in the first aspect. The graphical user interface training data includes a graphical user interface image, as well as the effective element region coordinates, fine-grained semantic labels, and interactive operation trajectory data corresponding to each interface element in the graphical user interface image. The graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interaction operation trajectory data are input into the graphical user interface agent model to be trained for model training, and a trained graphical user interface agent model is obtained. The trained graphical user interface (GUI) agent model is used to automatically generate corresponding operation decision results based on the interaction state of interface elements in the GUI image to be interactively controlled when a GUI image to be interactively controlled is received, so as to realize automated interactive control of the GUI.

[0018] According to a third aspect of the present disclosure, a graphical user interface training data generation apparatus is provided, comprising: The effective element region determination module is used to acquire the graphical user interface image to be processed and determine the coordinates of the effective element region corresponding to each interface element in the graphical user interface image. The element local image cropping module is used to crop the graphical user interface image locally according to the coordinates of the effective element region to obtain local sub-images corresponding to each of the interface elements. The interactive operation trajectory determination module is used to extract fine-grained semantic tags of each interface element through the local sub-image, and determine the interactive operation trajectory data corresponding to the graphical user interface image based on the fine-grained semantic tags. The training data construction module is used to generate the graphical user interface training data based on the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interactive operation trajectory data.

[0019] According to a fourth aspect of the present disclosure, a graphical user interface intelligent agent training device is provided, comprising: The training data acquisition module is used to acquire graphical user interface training data generated by the graphical user interface training data generation method in the first aspect. The graphical user interface training data includes a graphical user interface image, as well as the effective element region coordinates, fine-grained semantic labels and interactive operation trajectory data corresponding to each interface element in the graphical user interface image. The intelligent agent model training module is used to input the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels and the interaction operation trajectory data into the graphical user interface intelligent agent model to be trained, so as to obtain the trained graphical user interface intelligent agent model. The trained graphical user interface (GUI) agent model is used to automatically generate corresponding operation decision results based on the interaction state of interface elements in the GUI image to be interactively controlled when a GUI image to be interactively controlled is received, so as to realize automated interactive control of the GUI.

[0020] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement the methods as described in the first or second aspect.

[0021] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first or second aspect.

[0022] The technical solutions provided in this disclosure may have the following beneficial effects: The graphical user interface (GUI) training data generation method in the exemplary embodiments of this disclosure, on the one hand, determines the effective element region coordinates corresponding to each interface element in the GUI image, enabling each interface element in the GUI image to form a clear and quantifiable positional representation in space. This improves the consistency between the interface element positioning results and the actual display area, reduces sample errors caused by positional labeling deviations, and enhances the accuracy and stability of training data at the spatial labeling level, thereby improving the consistency and reliability of the training data. On the other hand, it performs local region cropping on the GUI image using the effective element region coordinates to obtain local sub-images corresponding to each interface element, and extracts fine-grained semantic tags from the local sub-images, thus making the semantic information extraction process more comprehensive. By focusing on the interface elements themselves, the accuracy of the association between semantic descriptions and corresponding interface elements is improved, enhancing the refinement of training data in terms of visual and interactive features, and improving the model's ability to recognize differences in interface element states and operable attributes. On the other hand, based on fine-grained semantic labels, interactive operation trajectory data corresponding to graphical user interface images are determined. This allows the training data to further incorporate sequential and process information of interactive behaviors, in addition to image information, effective element region coordinates, and fine-grained semantic labels. This improves the structured expression of training data at the interactive logic level, enhances the graphical user interface agent's ability to learn the mapping relationship between interface states and operational behaviors, and improves the rationality and stability of automated interactive decisions in multi-step scenarios.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0025] Figure 1 The illustration shows a flowchart of a graphical user interface training data generation method according to some embodiments of the present disclosure.

[0026] Figure 2 The illustration shows a flowchart of generating valid element region coordinates according to some embodiments of the present disclosure.

[0027] Figure 3 The illustration schematically depicts a process for constructing fine-grained semantic tags according to some embodiments of the present disclosure.

[0028] Figure 4 The illustration shows a flowchart of constructing interactive operation trajectory data according to some embodiments of the present disclosure.

[0029] Figure 5 The illustration shows a flowchart illustrating a sequence of steps for generating interactive elements according to some embodiments of the present disclosure.

[0030] Figure 6 The illustration schematically shows a flowchart of a graphical user interface agent training method according to some embodiments of the present disclosure.

[0031] Figure 7 The illustration shows a schematic diagram of the composition of a graphical user interface training data generation apparatus according to some embodiments of the present disclosure.

[0032] Figure 8 The diagram illustrates the composition of a graphical user interface intelligent agent training apparatus according to some embodiments of the present disclosure.

[0033] Figure 9 The schematic diagram illustrates the structural schematic of a computer system of an electronic device according to some embodiments of the present disclosure.

[0034] Figure 10 A schematic diagram of a computer-readable storage medium according to some embodiments of the present disclosure is shown.

[0035] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0037] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0038] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0039] Furthermore, the accompanying drawings are for illustrative purposes only and are not necessarily drawn to scale. The block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0040] In this example embodiment, a method for generating graphical user interface training data is first provided. This method can be applied to terminal devices or servers. This embodiment is not limited to this, and the following description will use the server executing the method as an example. Figure 1 The illustration schematically shows a flowchart of a graphical user interface training data generation method according to some embodiments of the present disclosure. (Reference) Figure 1 As shown, the graphical user interface training data generation method may include the following steps: Step S110: Obtain the graphical user interface image to be processed, and determine the effective element region coordinates corresponding to each interface element in the graphical user interface image; Step S120: Crop the graphical user interface image locally according to the coordinates of the effective element region to obtain a local sub-image corresponding to each interface element; Step S130: Extract fine-grained semantic tags for each interface element from the local sub-image, and determine the interactive operation trajectory data corresponding to the graphical user interface image based on the fine-grained semantic tags; Step S140: Generate the graphical user interface training data based on the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interactive operation trajectory data.

[0041] According to the graphical user interface training data generation method in this example embodiment, on the one hand, by determining the effective element region coordinates corresponding to each interface element in the graphical user interface image, each interface element in the graphical user interface image forms a clear and quantifiable positional expression in the spatial level, thereby improving the consistency between the interface element positioning results and the actual display area, reducing sample errors caused by positional labeling deviations, and improving the accuracy and stability of training data at the spatial labeling level, thus improving the consistency and reliability of training data; on the other hand, by cropping the graphical user interface image locally using the effective element region coordinates, local sub-images corresponding to each interface element are obtained, and fine-grained semantic tags are extracted from the local sub-images, making the semantic information extraction process more comprehensive. By focusing on the interface elements themselves, the accuracy of the association between semantic descriptions and corresponding interface elements is improved, enhancing the refinement of training data in terms of visual and interactive features, and improving the model's ability to recognize differences in interface element states and operable attributes. On the other hand, based on fine-grained semantic labels, interactive operation trajectory data corresponding to graphical user interface images are determined. This allows the training data to further incorporate sequential and process information of interactive behaviors, in addition to image information, effective element region coordinates, and fine-grained semantic labels. This improves the structured expression of training data at the interactive logic level, enhances the graphical user interface agent's ability to learn the mapping relationship between interface states and operational behaviors, and improves the rationality and stability of automated interactive decisions in multi-step scenarios.

[0042] The graphical user interface training data generation method in this example embodiment will be further described below.

[0043] In step S110, the graphical user interface image to be processed is acquired, and the coordinates of the effective element regions corresponding to each interface element in the graphical user interface image are determined.

[0044] In one example embodiment of this disclosure, a graphical user interface image refers to image data that can characterize the content currently displayed by the graphical user interface. For example, a graphical user interface image can be a bitmap image obtained by taking a screenshot from a terminal device, a frame image obtained by capturing a remote desktop screen, or an image exported from the rendering cache during application runtime. This embodiment does not impose any special limitation on the source of the graphical user interface image, as long as it can reflect the visual presentation of the interface elements.

[0045] Interface elements refer to display objects in a graphical user interface that have independent visual boundaries and carry interactive intent. For example, interface elements can be buttons, input boxes, icons, menu items, switch controls, etc. in a graphical user interface. This embodiment does not limit the type of interface elements.

[0046] Effective element region coordinates refer to the coordinate information used in a graphical user interface image to characterize the actual spatial location range of each interface element. For example, effective element region coordinates can be represented by a rectangle as a combination of the coordinates of the top left and bottom right corners, or by a combination of the center point coordinates and width and height parameters. Alternatively, a set of polygon vertex coordinates can be used to describe irregularly shaped interface element regions. This embodiment does not impose any special limitations on the specific expression of effective element region coordinates, as long as it uniquely determines the spatial range of the interface element in the graphical user interface image. Effective element region coordinates can indicate that the region coordinates corresponding to the interface element are not unverified initial detection results, but rather region coordinates determined after matching, filtering, boundary correction, or consistency verification, and are suitable for subsequent processing. For example, effective element region coordinates can be region coordinates generated after satisfying a preset overlap threshold between the initial element region coordinates and the candidate mask image, or region coordinates obtained by combining the initial element region coordinates with the region segmentation results and performing boundary correction. This embodiment is not limited to these limitations.

[0047] By determining the effective element region coordinates corresponding to each interface element in the graphical user interface image, each interface element in the graphical user interface image forms a clear and quantifiable positional expression in the spatial level. This improves the consistency between the interface element positioning results and the actual display area, reduces sample errors caused by positional labeling deviations, and enhances the accuracy and stability of training data at the spatial labeling level, thereby improving the consistency and reliability of training data.

[0048] In step S120, the graphical user interface image is cropped locally according to the coordinates of the effective element region to obtain local sub-images corresponding to each interface element.

[0049] In one example embodiment of this disclosure, local region cropping refers to extracting a corresponding image region from a graphical user interface image based on the coordinates of the effective element region, forming a local image slice centered on the interface element. For example, local region cropping can be performed by indexing and slicing the image matrix using an image processing library, such as extracting pixel data within a specified row and column range based on a two-dimensional array; alternatively, region extraction can be completed using a cropping function in an image processing framework. This embodiment does not limit the implementation method of local region cropping, as long as it can generate a local sub-image of the corresponding interface element.

[0050] A local sub-image refers to image data cropped based on the coordinates of the effective element region. It at least includes the main display area of ​​the interface elements, and in some optional implementations, it may also include a certain range of adjacent background areas. The local sub-image can retain the same resolution ratio as the original graphical user interface image, or it can be normalized in size according to the model input requirements, such as being uniformly scaled to a fixed size to adapt to the subsequent visual language model input. This embodiment does not limit whether scaling or normalization processing is performed.

[0051] Optionally, before cropping a local area, the boundary validity of the valid element area coordinates can be checked. For example, it can be determined whether the cropping area exceeds the boundary of the graphical user interface image. If there is an out-of-bounds situation, the cropping area can be truncated or filled to ensure that the generated local sub-image is valid image data.

[0052] By cropping the graphical user interface image locally based on the coordinates of the effective element regions, each interface element can generate a corresponding local sub-image, thus achieving image separation processing at the interface element level. This method can divide the global interface image into several local image units focused on a single interface element, which is beneficial for the subsequent fine-grained semantic tag extraction process to focus on the target interface element itself, reduce background interference, and improve the targeting and stability of semantic analysis.

[0053] In step S130, fine-grained semantic tags of each interface element are extracted from the local sub-image, and interactive operation trajectory data corresponding to the graphical user interface image are determined based on the fine-grained semantic tags.

[0054] In one exemplary embodiment of this disclosure, fine-grained semantic tags refer to a set of structured semantic information used to provide a refined description of the visual and interactive attributes of each interface element in a graphical user interface image. For example, fine-grained semantic tags may include information such as color features, brightness features, shape features, and interaction features, used to characterize the specific performance of interface elements in terms of visual presentation and operability. Of course, fine-grained semantic tags may also include other types of semantic information such as text content features, icon category features, functional attribute features, or hierarchical relationship features. This embodiment does not impose special limitations on the types of data features included in the fine-grained semantic tags, as long as they can reflect the visual differences and interactive states of interface elements.

[0055] Fine-grained semantic tags can be represented in a structured manner using key-value pairs, for example, stored in the form of "feature type - feature value", or they can be represented in a vectorized manner, encoding various visual and interactive features into semantic vectors of a unified dimension for use as input for model training. This embodiment does not impose any special restrictions on the data structure form of fine-grained semantic tags.

[0056] Fine-grained semantic tags can be generated by parsing local sub-images using a visual language model. For example, under the guidance of multi-dimensional prompt questions, the output can be response data targeting the features of interface elements, and the prompt questions and response data can be combined to form corresponding fine-grained semantic tags. Of course, rule-based image analysis methods or feature recognition methods based on classification models can also be used to generate fine-grained semantic tags. This embodiment does not impose any special limitations on the generation method of fine-grained semantic tags.

[0057] Interactive operation trajectory data refers to a set of information describing a series of ordered interactive actions performed in a graphical user interface image to achieve a preset target interactive result. For example, interactive operation trajectory data may include action types such as click, long press, swipe, drag, and text input, as well as corresponding interface element identification information, operation sequence information, and expected state change information. Of course, interactive operation trajectory data may also include other types of data such as operation time interval information, operation coordinate information, or operation result feedback information. This embodiment does not specifically limit the specific fields included in the interactive operation trajectory data, as long as they can completely describe the interactive process to achieve the target interactive result.

[0058] In optional implementations, the interactive operation trajectory data can be stored in the form of a structured list, such as recording each interactive operation step as an action sequence array; it can also be represented in the form of a graph structure, with interface elements as nodes and interactive actions as edges for association; or it can be represented in a vectorized manner, encoding the entire interactive step sequence into a sequence vector of fixed dimensions for model training input. This embodiment does not impose any special limitations on the specific data structure of the interactive operation trajectory data.

[0059] By constructing interactive operation trajectory data, the training data, which already includes image information, effective element region coordinates, and fine-grained semantic labels, further incorporates sequential and process information of interactive behaviors. This improves the structured representation of the training data at the interactive logic level, enhances the graphical user interface agent's ability to learn the mapping relationship between interface states and operational behaviors, and improves the rationality and stability of automated interactive decisions in multi-step scenarios.

[0060] In step S140, the graphical user interface training data is generated based on the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interactive operation trajectory data.

[0061] In one exemplary embodiment of this disclosure, graphical user interface (GUI) training data refers to a set of data samples used to train a GUI agent model. For example, GUI training data may include GUI images, effective element region coordinates corresponding to each interface element in the GUI image, fine-grained semantic labels, and interaction trajectory data. The GUI image reflects the overall visual content of the interface; the effective element region coordinates determine the spatial distribution of each interface element in the GUI image; the fine-grained semantic labels describe the semantic information of the interface elements in terms of color features, brightness features, morphological features, and interaction features; and the interaction trajectory data depicts the sequence of interaction steps required to achieve a specific target interaction result. Thus, the GUI training data can simultaneously represent interface visual information, spatial location information, semantic attribute information, and interaction flow information. Of course, GUI training data may also include auxiliary fields such as sample identification information, interface category identification information, task scenario identification information, or data version information. This embodiment does not specifically limit the types of fields included in the GUI training data, as long as they can support the training of the GUI agent model.

[0062] Graphical user interface (GUI) training data can be stored in a structured data format. For example, each training sample can be encapsulated in JavaScript Object Notation (JSON) format. Alternatively, it can be managed uniformly using a relational or non-relational database. Image files and annotation files can be stored separately and associated with unique identifiers. This embodiment does not impose any special limitations on the storage format of the GUI training data. In optional implementations, the GUI training data can be grouped or organized in batches according to interface scene type, functional module category, or interaction task type to adapt to different model training strategies, such as supervised learning training or policy learning training.

[0063] In some optional implementations, when generating graphical user interface training data, consistency checks can be performed on various types of data. For example, check whether the coordinates of the valid element region match the size of the graphical user interface image, check whether the fine-grained semantic labels correspond one-to-one with the corresponding interface elements, and check whether the interface element identifiers in the interactive operation trajectory data can find matching items in the coordinates of the valid element region to ensure the consistency of the internal logical relationship of the training data. This embodiment does not impose special limitations on the consistency check method, as long as the data structure is complete and the association is correct.

[0064] By linking and integrating graphical user interface (GUI) images, effective element region coordinates, fine-grained semantic labels, and interactive operation trajectory data, GUI training data is generated. This ensures that training samples simultaneously possess overall visual information of the interface, spatial location information of interface elements, semantic attribute information of interface elements, and interactive behavior flow information, thereby improving the structural integrity of the training data in terms of visual expression, spatial expression, semantic expression, and operational logic expression. By establishing a consistent mapping relationship between GUI images and corresponding effective element positions, fine-grained semantic labels, and interactive operation trajectory data within the same data sample, a unified data structure is formed for information of different dimensions, improving the accuracy of association between multimodal information and the consistency of sample organization, and enhancing the usability and scalability of the training data. By introducing interactive operation trajectory data as dynamic behavioral information, the training data not only reflects the static attributes of interface elements but also embodies the operational sequence and process constraints required to achieve the target interaction result, thereby improving the expressive ability of the GUI training data for the interface interaction process and enhancing the learning effect and decision-making stability of the GUI intelligent agent model in complex interaction scenarios.

[0065] The following is a detailed explanation of the graphical user interface training data generation method in steps S110 to S140.

[0066] In one example embodiment of this disclosure, it can be achieved through Figure 2 The steps in the process determine the coordinates of the effective element regions corresponding to each interface element in the graphical user interface image, referring to... Figure 2 As shown, it can specifically include: Step S210: Perform interface element detection on the graphical user interface image to obtain the initial element region coordinates of each interface element; Step S220: Perform region segmentation processing on the graphical user interface image based on the initial element region coordinates to obtain candidate mask images for each interface element; Step S230: Based on the matching degree between the initial element region coordinates and the candidate mask image, select a valid mask image from the candidate mask images, and generate the valid element region coordinates through the valid mask image.

[0067] Interface element detection refers to performing a global analysis of the entire graphical user interface image to identify candidate regions that may correspond to interface elements and output the initial element region coordinates corresponding to the candidate regions. Optionally, interface element detection can be implemented based on a deep learning object detection model. For example, a Convolutional Neural Network (CNN) or an object detection model based on a Vision Transformer (ViT) structure can be used. Alternatively, a detection model based on the Transformer architecture can be used to extract features from the graphical user interface image and predict the initial element region coordinates of each interface element. Of course, interface element detection can also be implemented by combining traditional image processing methods with rule matching methods, such as identifying interface element regions based on edge detection, color segmentation, or template matching. This embodiment does not impose any special limitations on the interface element detection method.

[0068] Initial element region coordinates refer to the predicted position of elements from a global perspective, used to represent the approximate spatial coordinate range of interface elements within a graphical user interface image. For example, initial element region coordinates can be represented by a rectangular bounding box, including the coordinates of the top-left and bottom-right corners; they can also be represented by the center point coordinates along with width and height parameters; or they can be represented by a set of polygon coordinates composed of multiple vertices. Any method that can characterize the spatial range of the area where the interface element is located is acceptable. This implementation does not impose any special limitations on the form of expression for initial element region coordinates.

[0069] Optionally, before detecting interface elements in the graphical user interface (GUI) image, image preprocessing can be performed. For example, operations such as size normalization, color space conversion, or contrast enhancement can be performed to improve the stability of subsequent detection. Then, interface elements can be identified based on color distribution features, edge contour features, texture features, or layout structure features in the image. For example, multi-scale feature maps can be extracted using a convolutional neural network, and candidate region generation and classification regression operations can be performed on the feature maps to output the initial element region coordinates corresponding to each interface element. Alternatively, rule-based connected component analysis, edge detection, or template matching methods can be used to detect interface elements, as long as the initial element region coordinates corresponding to each interface element can be output. This embodiment does not limit the specific implementation structure of the interface element detection algorithm. Through interface element detection, the preliminary determination of the positions of interface elements in the GUI image can be achieved.

[0070] Region segmentation refers to the process of distinguishing between interface elements and the background in a graphical user interface image, under the spatial constraints defined by the initial element region coordinates, to obtain pixel-level representations of the interface elements. The purpose of region segmentation is to further refine the true boundary range of interface elements based on the coarse spatial localization provided by the initial element region coordinates, thereby generating candidate mask images. It is understandable that region segmentation does not limit the specific algorithm structure, as long as it can output pixel-level mask regions corresponding to the interface elements under the constraints of the initial element region coordinates.

[0071] In some implementations, region segmentation can be achieved using deep learning-based segmentation methods. For example, semantic segmentation networks or instance segmentation networks can be used to perform pixel-level classification of graphical user interface images; alternatively, candidate regions of interest (ROIs) can be constructed within the regions defined by the initial element region coordinates, and segmentation network inference can be performed within these regions to obtain candidate mask images corresponding to the interface elements. The segmentation network can be built based on a convolutional neural network structure or a visual transformer structure; this embodiment does not limit this approach.

[0072] In other implementations, region segmentation can be achieved using a cue-guided approach, which involves converting the initial element region coordinates into point cue, bounding box cue, or region cue, constraining image features, and guiding the segmentation model to perform pixel-level mask prediction near the cue region.

[0073] In other embodiments, region segmentation can be implemented using traditional image segmentation algorithms. For example, edge detection-based segmentation methods can be performed within the region defined by the initial element region coordinates, determining the boundaries of interface elements by detecting pixel gradient changes; alternatively, a region growing method can be used, using the region corresponding to the initial element region coordinates as the seed region, and gradually expanding the pixel set based on color similarity or texture consistency to generate interface element regions; graph cut algorithms, GrabCut algorithms, or minimum cut optimization methods can also be used to optimize and obtain candidate mask regions under foreground and background cue constraints; alternatively, color clustering or connected component analysis methods can be used to classify pixels within the initial region and extract the main connected components as interface element regions. This disclosure does not limit the specific segmentation algorithm.

[0074] In other implementations, region segmentation can also be achieved in a hybrid manner. For example, a preliminary mask result can be obtained first based on a deep learning model, and then the mask boundary can be optimized by combining morphological processing; or the contour can be obtained first based on edge detection, and then the boundary can be refined by learning a model.

[0075] A candidate mask image refers to image data used to represent the pixel-level range of interface elements. For example, a candidate mask image can be a binary image, where the pixel values ​​of the interface element region are the first value and the pixel values ​​of the background region are the second value; it can also be a probabilistic image, where each pixel value represents the probability that the pixel belongs to an interface element. This implementation does not impose any special limitations on the representation format of the candidate mask image. Region segmentation processing can be implemented using graph cut-based algorithms, region growing-based algorithms, clustering-based algorithms, or deep learning semantic segmentation networks, as long as it can output candidate mask images corresponding to each interface element. Through region segmentation processing, a more refined representation of the interface element boundaries than the initial element region coordinates can be obtained.

[0076] After obtaining the candidate mask images, valid mask images are selected from the candidate mask images based on the matching degree between the initial element region coordinates and the candidate mask images. Valid element region coordinates are then generated from the valid mask images. The matching degree refers to a numerical index used to measure the consistency between the region represented by the initial element region coordinates and the interface element regions in the candidate mask images. For example, the Intersection over Union (IoU) value between the region covered by the initial element region coordinates and the interface element regions in the candidate mask images can be calculated to measure the matching degree; alternatively, the pixel overlap ratio or boundary distance error between the two can be calculated to measure the matching degree, as long as it reflects the spatial consistency between the two. This embodiment does not impose any special limitations on the calculation method of the matching degree.

[0077] A valid mask image is a candidate mask image that, under preset matching conditions, is determined to accurately reflect the actual boundary range of interface elements. For example, candidate mask images with a matching degree greater than or equal to a preset threshold can be considered valid mask images.

[0078] After selecting the valid mask images, the coordinates of the valid element regions can be generated based on the set of pixel values ​​of the interface elements in the valid mask images. For example, the minimum bounding rectangle of the interface element regions in the valid mask images can be extracted, or its minimum bounding box can be calculated, and the coordinates of the bounding region can be used as the coordinates of the valid element regions; alternatively, the coordinates of the valid element regions in the form of a polygonal coordinate set can be directly generated based on the mask outline, and this implementation is not limited to this.

[0079] By detecting interface elements in graphical user interface images and generating candidate mask images through region segmentation, and then selecting valid mask images based on the matching degree to determine the coordinates of valid element regions, the positioning results of interface elements have both global recognition capability and local boundary accuracy, thereby improving the accuracy and stability of element region determination and enhancing the reliability of subsequent training data in spatial representation.

[0080] In one example embodiment of this disclosure, the following steps can be used to filter valid mask images from candidate mask images based on the degree of matching between the initial element region coordinates and the candidate mask images. Specifically, these steps may include: It can calculate the degree of overlap between the initial element region coordinates and the candidate mask image; and select candidate mask images with an overlap degree greater than or equal to a preset overlap degree threshold as valid mask images.

[0081] The degree of overlap can be measured by the intersection-union ratio, which is the ratio of the intersection area to the union area of ​​the initial region pixel set and the candidate mask pixel set. Alternatively, the degree of overlap can be measured by the pixel overlap ratio, which is the overlap ratio between the pixels in the initial element region coordinates and the pixels in the candidate mask image. Of course, the degree of overlap can also be measured by the ratio of the intersection area to the area of ​​the initial region, the ratio of the intersection area to the area of ​​the candidate mask region, etc., as long as it can reflect the spatial coverage relationship between the two. This embodiment does not limit the specific measurement index of the degree of overlap.

[0082] In some optional implementations, to improve the stability of overlap calculation, mask preprocessing operations can be performed on the candidate mask images before calculation. These operations could include hole filling, morphological opening and closing operations, or small connected component removal to reduce the impact of noisy pixels on the overlap degree. Alternatively, the boundaries of the initial region pixel set can be slightly expanded or contracted to adapt to boundary deviations at different detection scales. This implementation does not limit the preprocessing method. If the same interface element corresponds to multiple candidate mask images, the overlap degree can be calculated separately for each candidate mask image to form a one-to-one correspondence between candidate mask images and overlap degrees.

[0083] Candidate mask images with an overlap degree greater than or equal to a preset overlap degree threshold can be used as valid mask images. That is, based on the aforementioned overlap degree measurement results, the candidate mask images are filtered using threshold judgment rules, and candidate mask images that meet the spatial consistency requirements are retained as valid mask images, so that the mask results used subsequently are sufficiently consistent with the coordinates of the initial element region.

[0084] The preset overlap threshold is a threshold parameter used to determine whether the candidate mask image meets the consistency requirements. It can be a fixed value threshold or an adaptive threshold set according to factors such as the resolution of the graphical user interface image, the size distribution of interface elements, or the complexity of the candidate mask area. It can be customized according to the actual use scenario. This embodiment does not limit the value of the preset overlap threshold.

[0085] During the filtering process, when the overlap degree of a candidate mask image is greater than or equal to a preset overlap degree threshold, the candidate mask image can be directly determined as a valid mask image; when the overlap degree is less than the preset overlap degree threshold, the candidate mask image can be discarded to avoid introducing mask results that are inconsistent with the initial element region coordinates into subsequent processing. This embodiment does not limit the number of valid mask images after filtering. For example, multiple valid mask images can be allowed for the same interface element, or when multiple candidate mask images that meet the conditions exist, the candidate mask image with the highest overlap degree can be selected as the valid mask image.

[0086] By calculating the degree of overlap between the initial element region coordinates and the candidate mask image, and filtering the effective mask image based on a preset overlap threshold, the element region filtering process is equipped with a quantitative judgment standard and a geometric consistency control mechanism. This improves the controllability and repeatability of the generation of effective element region coordinates and enhances the stability of the training data construction process. At the same time, filtering the effective mask image by the degree of overlap can effectively ensure the geometric consistency between the initial element region coordinates and the candidate mask image, thereby improving the accuracy and stability of element region positioning.

[0087] In an optional embodiment of this disclosure, interface element detection of a graphical user interface image can be achieved by the following steps to obtain the initial element region coordinates of each interface element, specifically including: The graphical user interface image can be input into the trained point prediction multimodal model to obtain the initial element region coordinates of each interface element. The point prediction multimodal model is used to perform global detection on the graphical user interface image.

[0088] Among them, a well-trained point prediction multimodal model refers to a multimodal model that has been trained using pre-constructed training data and can output the spatial point or region location of interface elements based on visual information. A point prediction multimodal model is a multimodal detection model capable of performing point-level or region-level localization prediction of multiple target elements in an image within a global visual context. For example, a point prediction multimodal model could be the Rex-Omni model. The Rex-Omni model can achieve initial localization of interface elements. Built on a multimodal fusion architecture, the Rex-Omni model integrates semantic information of the interface structure during visual feature encoding, maintaining high localization accuracy under complex layout conditions. Especially in detection scenarios with a large number of dense objects, the Rex-Omni model can stably output a set of candidate points with high confidence, thereby improving the recognition performance in scenarios with dense interface elements.

[0089] Point prediction multimodal models can locate elements by focusing on points when performing global detection on graphical user interface images. For example, they can predict the center point, key point, or anchor point of interface elements, and then deduce the corresponding area range based on the predicted points. In this way, interference caused by overlapping anchor frames in dense scenes can be reduced, and the ability to distinguish small and closely arranged interface elements can be improved. Especially in PC graphical user interface scenarios, interface elements are characterized by a large number and compact layout. The point prediction mechanism of point prediction multimodal models can improve the separation accuracy between interface elements and reduce missed detections and duplicate detections.

[0090] When performing global detection on graphical user interface images, the point prediction multimodal model scans the positions of interface elements and identifies candidate regions across the entire image range, rather than judging only local regions. This allows it to capture the spatial distribution relationships and layout patterns between interface elements. This global detection capability helps generate more stable and accurate initial element region coordinates.

[0091] For example, after developing a multimodal model for predicting image input points in a graphical user interface, features can first be extracted using a visual encoder to obtain an image feature tensor. Then, a prediction head network performs position regression and confidence prediction on the feature tensor, outputting several candidate points and their corresponding region parameters. Next, initial element region coordinates can be generated based on the candidate points and region parameters. The initial element region coordinates can be represented using a rectangle or by a center point and width and height parameters, as long as they can represent the spatial extent of the interface elements.

[0092] In this embodiment, the initial element region coordinates output by the point prediction multimodal model are not only used for the initial positioning of interface elements, but also serve as the benchmark input for subsequent geometric consistency matching calculations. Since the candidate mask image needs to be matched with the initial element region coordinates, the accuracy of the initial element region coordinates directly affects the stability of the geometric consistency screening stage. Using the point prediction multimodal model to generate relatively stable initial element region coordinates can reduce the probability of mismatch and improve the overall reliability and accuracy of generating effective element region coordinates.

[0093] In some alternative implementations, to improve the stability of the initial element region coordinates, the output of the point prediction multimodal model can be post-processed, for example, by performing non-maximum suppression (NMS) to remove duplicate prediction boxes with high overlap; of course, low-confidence prediction results can also be filtered based on a confidence threshold. This implementation does not limit the post-processing method.

[0094] By inputting graphical user interface images into a trained point prediction multimodal model for global detection, the initial element region coordinates of each interface element are output. This enables the recognition process of interface elements to integrate visual features and semantic information for comprehensive judgment, thereby improving the generalization ability and recognition completeness of interface element detection in complex interface scenarios.

[0095] In an optional embodiment of this disclosure, the graphical user interface image can be segmented based on the initial element region coordinates to obtain candidate mask images for each interface element through the following steps, specifically including: The graphical user interface image and the initial element region coordinates can be input into the trained cue-driven segmentation model to obtain candidate mask images of each interface element. The cue-driven segmentation model is used to perform element mask recognition on the graphical user interface image with the initial element region coordinates as the cue target.

[0096] The cue-driven segmentation model refers to a model structure capable of finely segmenting a target region under the guidance of cue information. For example, the cue-driven segmentation model can be a Segment Anything Model (SAM) or a segmentation network built based on an encoder-decoder structure; this embodiment is not limited to these. In an optional implementation, the cue-driven segmentation model may include a visual encoder and a mask prediction decoder. The visual encoder is used to extract global feature representations of the graphical user interface image, and the decoder combines the cue information to generate corresponding candidate mask images. This embodiment does not limit the specific network structure of the cue-driven segmentation model, as long as it can output candidate mask images corresponding to interface elements based on the initial element region coordinates.

[0097] In this embodiment, the initial element region coordinates can be converted into cue signals that the cue-driven segmentation model can recognize, such as point cue, bounding box cue, or region cue. For example, when the initial element region coordinates are a rectangular bounding box, the bounding box can be used as a bounding box cue input to the model; when the initial element region coordinates are a center point, the center point can be used as a positive sample cue point input to the model; of course, multiple points can also be combined into a multi-point cue form and input to the model, and this embodiment is not limited thereto.

[0098] In one optional implementation, the graphical user interface image can first be feature-extracted by the visual encoder in the cue-driven segmentation model to generate an image feature map; then, the cue information corresponding to the initial element region coordinates is encoded into a cue embedding vector and fused with the image feature map; next, the decoder generates a candidate mask image based on the fused features.

[0099] In some alternative implementations, to improve the stability of the candidate mask image, morphological processing, such as erosion or dilation operations, can be performed on the candidate mask image to eliminate noisy pixels; alternatively, the probability mask can be converted into a binary mask based on a probability threshold.

[0100] By inputting the graphical user interface image and the initial element region coordinates into the trained prompt-driven segmentation model for element mask recognition, the candidate mask image can more accurately fit the real contour of the interface element, thereby improving the fineness of the boundary representation of the interface element region and enhancing the spatial accuracy of the effective element region coordinates.

[0101] In one example embodiment of this disclosure, the following steps can be used to crop a graphical user interface image locally based on the coordinates of the effective element region to obtain a local sub-image corresponding to each interface element. Specifically, this may include: The initial cropping region can be determined based on the coordinates of the effective element region, and the target cropping region can be obtained by expanding the region outward along the periphery of the initial cropping region with a preset width ratio; the graphical user interface image is cropped according to the target cropping region to obtain local sub-images corresponding to each interface element.

[0102] The initial cropping region refers to the image region range directly mapped from the coordinates of the effective element region, which corresponds to a sub-region in the graphical user interface image at the pixel level. Specifically, the horizontal and vertical starting pixels, as well as the horizontal and vertical spans, can be determined in the pixel coordinate system of the graphical user interface image based on the coordinates of the effective element region, thereby cropping the corresponding rectangular region in the image buffer. Alternatively, in an optional implementation, a polygonal cropping region can be constructed based on the coordinates of the effective element region. This implementation does not strictly limit the geometric shape of the cropping region. It is understood that, for ease of explanation, a rectangular region will be used as an example in the following descriptions.

[0103] Expanding a region outwards along the periphery of the initial cropping area by a preset width ratio refers to expanding the width and height of the region according to a preset width ratio based on the boundary of the initial cropping area. The preset width ratio can be a proportional value relative to the width or height of the initial cropping area, such as expanding by 5%, 10%, or other ratios of the initial cropping area width; alternatively, a fixed pixel value can be used as the expansion width. This embodiment does not limit the value of the preset width ratio, as long as it forms an additional context area outside the initial cropping area.

[0104] In the specific implementation process, the left, right, top, and bottom boundaries of the initial cropping area can be expanded proportionally or unequally. For example, the area can be uniformly expanded at the left, right, top, and bottom boundaries of the initial cropping area according to a preset width ratio. Alternatively, it can be expanded to the left, right, up, or down by the same or different number of pixels corresponding to the preset width ratio, thereby forming a new area boundary. Simultaneously, the boundary validity can be verified during the expansion process to prevent the expanded target cropping area from exceeding the actual size range of the graphical user interface image. When the expanded area exceeds the image boundary, truncation can be performed to keep the target cropping area within the effective pixel range of the graphical user interface image. It is understood that this is merely an illustrative example and is not intended to be limiting.

[0105] The graphical user interface (GUI) image can be cropped based on the target cropping region. This involves extracting corresponding image data from the GUI image according to the spatial range defined by the target cropping region, generating a local sub-image. A local sub-image refers to an image fragment containing only the corresponding interface element and its surrounding context area. This can be achieved at the pixel level through image array slicing operations or by calling a cropping interface from an image processing library. This implementation does not limit the specific implementation method of the cropping operation.

[0106] The local sub-images obtained through the above steps form a one-to-one correspondence with each interface element, so that each interface element corresponds to an image area containing its main visual content and appropriate contextual information. By setting a preset width ratio to expand the initial cropping area, it is possible to avoid the loss of semantic information due to excessive cropping while ensuring local focus. This improves the matching degree between semantic analysis results and the actual interface presentation, and enhances the stability and information completeness of subsequent fine-grained semantic tag extraction.

[0107] In one example embodiment of this disclosure, it can be achieved through Figure 3 The steps described in the document enable the extraction of fine-grained semantic tags for each interface element from local sub-images, as referenced. Figure 3 As shown, it can specifically include: Step S310: Perform multi-dimensional semantic analysis on the local sub-image to generate multi-dimensional prompt questions targeting the interface element features in the local sub-image; Step S320: Generate interface element feature answer data corresponding to the local sub-image based on the multi-dimensional prompt question and the local sub-image; Step S330: The multi-dimensional prompt question and the interface element feature answer data are used as fine-grained semantic tags for interface elements in the local sub-image.

[0108] Multi-dimensional semantic analysis refers to the process of performing structured semantic parsing on interface elements within a local sub-image from multiple attribute dimensions. For example, multi-dimensional semantic analysis can include a comprehensive analysis of the visual attributes, content attributes, and interaction state attributes of interface elements to obtain a fine-grained semantic expression of the interface elements in the current interface state. The purpose of multi-dimensional semantic analysis is to decompose the semantic understanding of interface elements into multiple structured dimensions, rather than directly outputting a single descriptive text. By breaking down the semantic understanding process into a two-stage process of "question generation—question answering," it can explicitly guide semantic analysis to cover multiple attribute dimensions, reducing semantic omissions that may occur due to direct description.

[0109] In some implementations, multi-dimensional prompts can be constructed around different attributes of interface elements. For example, questions about visual attributes can be generated, including color features, brightness features, boundary shape features, or texture features; questions about content attributes can be generated, such as text content, icon category, or symbol meaning; and questions about interactive attributes can also be generated, such as whether it is active, disabled, selected, or highlighted. This disclosure does not limit the specific types and number of prompts, as long as they can cover multiple semantic dimensions of the interface elements.

[0110] The generation of multi-dimensional prompt questions can employ various implementation mechanisms. For example, a question framework can be constructed based on a preset attribute dimension template, and then specific question text can be generated by combining visual features in local sub-images; alternatively, a visual language model can be used to directly generate question text based on local sub-images; or a rule-driven approach can be used to automatically generate corresponding attribute questions based on the type of interface elements. This disclosure does not limit the generation method of multi-dimensional prompt questions.

[0111] After generating multi-dimensional prompts, interface element feature response data can be generated based on the prompts and local sub-images. This interface element feature response data can be used to provide specific attribute values ​​or state judgments for each prompt. For example, in a question about color features, the interface element feature response data could be "dark blue" or "light gray"; in a question about interaction status, it could be "clickable" or "disabled"; and in a question about text content, it could be the corresponding character content. It is understood that this embodiment does not limit the expression form of the interface element feature response data; it can be natural language text or a structured attribute value representation.

[0112] The generation of interface element feature response data can be implemented in various ways. For example, a visual language model can be used to understand local sub-images and generate response text; an image classification model can be used to identify different attribute categories and then map the identification results to response content; a rule-based judgment method can also be used, such as determining whether a button is disabled by using a pixel color threshold. This embodiment does not limit the generation method of interface element feature response data.

[0113] After generating the prompt questions and corresponding answer data, the question text and answer content can be associated and combined to form one or more question-answer pairs. Each question-answer pair corresponds to an attribute dimension of the interface element, and the collection of multiple question-answer pairs constitutes the fine-grained semantic tag of the interface element.

[0114] At the data structure level, fine-grained semantic tags can be represented as a set of key-value pairs or as an array of question-answer pairs. For example, it can be represented in the following structured way: {"question": "Is this button currently clickable?", "answer": "Yes, currently highlighted and clickable"}; multiple question-answer pairs can be integrated into a unified data structure, such as a JSON object or list, for use in subsequent training data construction.

[0115] By adopting a semantic generation approach of "generating questions first, then generating answers," the semantic analysis process can explicitly cover multiple attribute dimensions, thus avoiding the attribute omissions or ambiguities that may occur with direct holistic descriptions. Simultaneously, since semantic tags are expressed in a structured question-and-answer pair format, the correspondence between semantic information and interface elements can be enhanced, improving the interpretability and consistency of training data at the semantic expression level. Furthermore, the multi-dimensional prompting question generation mechanism can strengthen attention to subtle visual differences while focusing on local sub-images. For example, in scenarios where interface elements are small or their states change subtly, targeted question guidance can improve the accuracy of recognizing differences in brightness, color intensity, or boundary details, thereby enhancing the precision of fine-grained semantic tags.

[0116] In an optional embodiment, fine-grained semantic tags for each interface element can be extracted from local sub-images through the following steps, specifically including: Local sub-images can be input into a trained visual language model to generate multi-dimensional prompts targeting the interface element features in the local sub-images; the multi-dimensional prompts and local sub-images can be re-input into the visual language model to obtain interface element feature response data corresponding to the local sub-images; the multi-dimensional prompts and interface element feature response data can be used as fine-grained semantic labels for the interface elements in the local sub-images.

[0117] The visual language model refers to a multimodal model that can process visual and textual information simultaneously. For example, the visual language model can be a multimodal model structure built based on the Transformer architecture, which can include a visual encoder and a text decoder. The visual encoder is used to extract visual features from local sub-images, and the text decoder is used to generate corresponding language descriptions based on the visual features. Of course, the visual language model can also adopt a fusion encoding structure, that is, jointly modeling visual features and textual prompts in the same model, as long as it can achieve semantic understanding and language generation of image content. This implementation does not limit the specific network structure of the visual language model.

[0118] A well-trained visual language model refers to a model obtained by pre-training on large-scale image-text alignment data or fine-tuning on domain-specific data. Its parameters are fixed or partially fixed, and it is used to perform semantic analysis on input local sub-images. In specific implementations, the local sub-images can be pre-processed, such as performing size normalization, pixel value normalization, and color space conversion to adapt to the input format of the visual language model; alternatively, the local sub-images can be encoded into visual feature vectors before being input into the model. This implementation does not impose any special limitations on this approach.

[0119] A visual language model can be used to perform preliminary visual understanding of local sub-images and generate suggestive question text to guide further semantic mining. Multi-dimensional prompts can be constructed around different visual or interactive dimensions of interface elements. For example, multi-dimensional prompts may include questions about color attributes, morphological structures, text content, and interactive states (such as active / selected / disabled / highlighted). This implementation does not limit the specific content of the multi-dimensional prompts.

[0120] In generating multi-dimensional prompt questions, an autoregressive generation method can be used, that is, to gradually generate question text based on visual features; alternatively, a combination of template-driven and model-generated methods can be used. For example, several question templates can be preset, and then the visual language model can fill in the variables in the templates. This implementation does not limit the generation mechanism of the prompt questions. By generating multi-dimensional prompt questions, the semantic analysis of interface elements can be decomposed into multiple interpretable dimensions, which is beneficial for the subsequent formation of structured semantic tags.

[0121] The multi-dimensional prompt question and local sub-images can be re-input into the visual language model. That is, based on the existing visual input, the generated multi-dimensional prompt question is used as text input, forming a multimodal input pair with the local sub-images, and then input again into the visual language model for reasoning. At this point, after receiving the joint input of visual features and the question text, the visual language model generates conditions for the question, thereby outputting the answer to the question.

[0122] The visual language model can be guided by multi-dimensional prompts to output interface element feature response data corresponding to local sub-images. The visual language model can generate corresponding response text or structured description data based on each prompt. Interface element feature response data can be expressed in natural language, for example, describing the color category, brightness variation, boundary morphology, and activation status of interface elements; or it can be converted into structured field format, such as storing attribute names and values ​​as key-value pairs. This implementation does not limit the specific representation of the response data.

[0123] Each prompt question can be paired with its corresponding answer data and stored in a structured manner as fine-grained semantic tags. Fine-grained semantic tags refer to a set of attributes for a single interface element across multiple semantic dimensions. They include not only basic category information of the interface element but also detailed descriptions related to its visual appearance and interactive attributes. Fine-grained semantic tags can be saved in JSON, dictionary structure, or multi-field record format, as long as they can clearly express multi-dimensional semantic information. This implementation does not limit the storage format of the tags.

[0124] By inputting local sub-images into a visual language model, and generating interface element feature response data under the guidance of multi-dimensional prompts, the visual and interactive attributes of interface elements can be systematically extracted and expressed, thereby improving the richness and structure of fine-grained semantic labels and enhancing the ability of training data to characterize the state differences of interface elements.

[0125] Visual language models can establish cross-modal alignment between visual features and linguistic expressions, enabling the visual states of interface elements to be directly mapped into structured semantic representations. For example, when a button is displayed in light gray, the visual language model can not only recognize color features but also infer, based on semantic knowledge, that it may be disabled, thus generating a response such as "currently not clickable." This cross-modal reasoning capability enhances the depth of understanding of the operability of interface elements, going beyond mere surface-level visual descriptions.

[0126] Secondly, when generating question-and-answer pairs, visual language models can make comprehensive judgments by combining contextual semantic information. When a local sub-image contains only a single interface element, the visual language model can combine visual details with general interface semantic knowledge. For example, it can associate "the checkbox is selected" with "the function is enabled" to generate logically meaningful answers. This allows fine-grained semantic labels to not only describe appearance features but also implicitly contain operational logic information.

[0127] Visual language models possess a certain degree of expressive diversity and generalization ability when dealing with natural language problems. By guiding questions through multi-dimensional prompts, they can generate clearly structured and semantically complete answer texts, thereby enhancing the linguistic expressive diversity of training data. This helps subsequent intelligent agent models maintain consistent understanding under different semantic expressions and improves generalization ability.

[0128] Furthermore, visual language models can maintain good recognition stability in scenarios with subtle visual differences. When interface elements are small or have subtle color differences, combining language prompts with targeted analysis can enhance the model's focus on local visual features, thereby improving the recognition accuracy of brightness changes, edge changes, or icon detail differences, and enhancing the accuracy of fine-grained semantic labels.

[0129] Furthermore, since the question-answer pairs generated by the visual language model naturally possess a linguistic structure, they can be directly used as input conditions for subsequent trajectory planning by the large language model. For example, in the trajectory construction stage, preliminary operation steps can be automatically generated based on the answer "the current button is disabled." Because the semantic expression form has good interface consistency with the trajectory planning module, data conversion costs can be reduced and the overall data generation process can be made more coherent.

[0130] In this embodiment, the interface element features may include at least one of color features, brightness features, shape features, and interaction features; the interaction features may include an active state, a disabled state, a selected state, or a highlighted state.

[0131] The interface element features are not limited to a single category label, but rather a multi-dimensional set of attributes formed around the actual presentation effect of the interface elements in the graphical user interface image. This embodiment does not limit the specific number of dimensions included in the interface element features, as long as they belong to at least one of color features, brightness features, shape features, and interaction features.

[0132] Color features refer to information used to describe the color attributes of interface elements in their visual presentation. For example, color features may include the dominant color category, color distribution ratio, the presence of gradient effects, and the difference between border and fill colors. In practical implementation, color histograms can be generated based on the pixel distribution of interface element regions in local sub-images, or color semantics can be abstractly described using a visual language model, such as generating semantic tags like "blue button" and "red highlighted text." Alternatively, color features can be represented as numerical ranges in a standard color space, such as the Red Green Blue (RGB) color space or the Hue Saturation Value (HSV) color space. This implementation does not limit the representation method of color features. By introducing color features, the ability of fine-grained semantic tags to characterize the visual differences of interface elements can be enhanced.

[0133] Brightness features refer to information used to describe the brightness of interface elements in an image. For example, brightness features may include average brightness value, brightness contrast level, and whether it is in a highlighted display state. In specific implementations, brightness values ​​can be obtained by performing grayscale conversion on interface element regions in local sub-images and calculating the average grayscale value; alternatively, descriptive text can be generated using a visual language model, such as semantic expressions like "in a highlighted state" or "darker color." This implementation does not limit the calculation method or expression form of brightness features. By introducing brightness features, the ability of fine-grained semantic tags to express the visual hierarchy and state changes of interface elements can be improved.

[0134] Morphological features refer to information used to describe the spatial structure and geometric shape of interface elements. For example, morphological features can include the outline shape category of interface elements, such as rectangle, circle, ellipse, or irregular shape, and can also include geometric attributes such as the roundness of corners, border thickness, and aspect ratio. In practical implementation, the outline of interface elements can be extracted and shape parameters calculated using image processing algorithms, or the morphology of interface elements can be semantically described using a visual language model. For example, semantic tags such as "rounded corner button" and "slender input box" can be generated. This implementation does not limit the method of obtaining morphological features. By introducing morphological features, the ability of fine-grained semantic tags to express the structural characteristics of interface elements can be improved.

[0135] Interaction features refer to information describing the interactive state of interface elements. For example, interaction features may include active, disabled, selected, or highlighted states. An active state means the interface element is currently in a state where an action can be triggered, such as a button being clickable; a disabled state means the interface element is currently not in a state where an action can be triggered, such as a button being grayed out and unclickable; a selected state means the interface element is in a selected state, such as a checkbox being checked; a highlighted state means the interface element is highlighted due to gaining focus or mouse hover. This implementation does not limit the specific types of interaction features.

[0136] In the specific implementation process, interaction features can be inferred by analyzing changes in the color, brightness, or shape of interface elements, or by using a visual language model combined with contextual semantics. For example, when an interface element has a light color and reduced transparency, it can be determined to be in a disabled state; when an interface element has a thickened border or a brightened background, it can be determined to be in a highlighted state. The specific determination can be made according to the actual application scenario. This implementation does not limit the determination logic of interaction features.

[0137] By incorporating at least one of the color, brightness, shape, and interaction features into the feature set of interface elements, fine-grained semantic tags can characterize interface elements across multiple visual and state dimensions. This improves the comprehensiveness and precision of the semantic expression of interface elements, and enhances the accuracy of judging the operability of interface elements when constructing interactive operation trajectory data based on fine-grained semantic tags.

[0138] In one example embodiment of this disclosure, it can be achieved through Figure 4 The steps in the document implement the determination of interactive operation trajectory data corresponding to graphical user interface images based on fine-grained semantic tags, referencing... Figure 4 As shown, it can specifically include: Step S410: Determine the target interaction result corresponding to the graphical user interface image; Step S420: Determine the state conditions that at least some interface elements need to satisfy when achieving the target interaction result based on the fine-grained semantic tags. Step S430: Construct an interaction step sequence corresponding to at least some interface elements when the state conditions are met based on the fine-grained semantic tags, and use the interaction step sequence as the interaction operation trajectory data corresponding to the graphical user interface image.

[0139] The target interaction result refers to the expected interface change or functional achievement result obtained through a series of interactive operations in the current interface state presented by the graphical user interface image. The target interaction result can be understood as the expected state of the interface after a series of operations are performed. For example, the target interaction result may be that a certain function page is successfully opened, a certain setting is successfully modified, a certain button is triggered and generates a corresponding feedback result, etc. This embodiment does not limit the specific type of the target interaction result, as long as a verifiable result state can be formed at the interface state level.

[0140] In the specific implementation process, the target interaction result can be defined by preset rules. For example, the target state can be predefined according to the interface scene category; it can also be inferred based on the semantic information of interface elements contained in fine-grained semantic tags. For example, when there are elements such as "login button" and "account input box" in the graphical user interface, "complete login process" can be set as the target interaction result. This implementation does not limit the method of determining the target interaction result. By clarifying the target interaction result, directional constraints can be provided for the construction of subsequent interaction operation trajectory data.

[0141] Given that the target interaction result is already determined, the interface element state constraints that must be met to achieve the target interaction result can be derived by combining the information about the interface element attributes in the fine-grained semantic tags. For example, state conditions may include the active state, visible state, selected state, or specific visual performance state of the interface element. This implementation does not limit the specific types of state conditions.

[0142] Optionally, information about interaction features, color features, or morphological features in fine-grained semantic tags can be traversed to identify UI elements associated with the target interaction result. For example, when the target interaction result is "form submitted successfully," the submit button, which needs to be clickable, and the text input box, which needs to be in an entered state, can be identified, and the states corresponding to these UI elements can be used as state conditions. By introducing state conditions, a logical association can be established between the target interaction result and the current attributes of the UI elements.

[0143] It is possible to construct an interaction step sequence corresponding to at least some interface elements when state conditions are met based on fine-grained semantic tags. That is, given a clear state condition, a set of ordered interactive actions can be generated by combining the semantic attributes of interface elements recorded in the fine-grained semantic tags. An interaction step sequence refers to a set of operations arranged in a certain order, such as click, input, swipe, or selection operations.

[0144] In an optional implementation, interface elements that meet the state conditions can be selected first, and then corresponding interactive actions can be determined for each interface element. For example, a click action can be generated for an active button, and an input action can be generated for a text input box. These actions are then arranged in the logical order required to achieve the target interactive result. The sequence of interactive steps can be represented using a list structure, with each element corresponding to an interactive action and its target object. This implementation does not limit the specific representation of the sequence of interactive steps. The sequence of interactive steps can be used as interactive operation trajectory data corresponding to a graphical user interface image. The interactive operation trajectory data can include information such as interactive step identifiers, target interface element identifiers, and action types, which are used to represent the operation path from the initial interface state to the target interactive result.

[0145] By determining the target interaction result and combining fine-grained semantic label analysis to determine the state conditions that interface elements need to meet and the sequence of interaction steps, the interaction operation trajectory data can reflect the functional relationships and execution flow between interface elements, thereby improving the completeness and consistency of the training data at the interaction logic level.

[0146] In an optional embodiment of this disclosure, it can be achieved through Figure 5 The steps described in the document are implemented by constructing a sequence of interaction steps corresponding to at least some interface elements when state conditions are met, based on fine-grained semantic tags. (See reference...) Figure 5 As shown, it can specifically include: Step S510: Determine the state dependency relationship between at least some interface elements based on the fine-grained semantic tags, and determine the logical operation order corresponding to at least some of the interface elements according to the state dependency relationship. Step S520: Determine the interaction actions corresponding to at least some of the interface elements when the state conditions are met, and combine the interaction actions according to the logical operation sequence to form the interaction step sequence.

[0147] State dependency refers to the pre-constraints or triggering associations that exist in the state changes of different interface elements when achieving the target interaction result. It can be understood that the state change of one interface element will affect the state of another interface element. That is, the activation state of one interface element may depend on the completion of input or selection of another interface element before it can be triggered, or the display of a certain interface element depends on whether another interface element is clicked. For example, a button will only change from a disabled state to an active state when the input box is in the input state, or a new sub-menu interface will only be displayed after a certain option is selected.

[0148] In the specific implementation process, the descriptive information about interaction features, morphological features, or other attributes in the fine-grained semantic tags can be traversed to identify logically related pairs of interface elements. For example, when the fine-grained semantic tags indicate that a button is disabled, and a change in the state of another interface element may cause that button to become active, a dependency relationship can be established between the two. In an optional implementation, a state dependency graph structure can be constructed, with interface elements as nodes and dependencies as directed edges, thereby representing the logical constraints between interface elements in the form of a graph model. This implementation does not limit the way state dependencies are represented.

[0149] Given the identified dependencies between interface elements, the execution order of interactive operations can be sorted according to dependency constraints. The logical operation order can be understood as the sequential order in which different interface elements perform actions to achieve the desired interactive result. For example, if the activation state of a button depends on the content state of an input box, the input operation should precede the button click operation. Optionally, a topological sorting algorithm can be used to sort the state dependency graph, or a rule-based sequential derivation method can be used, as long as the dependencies are satisfied. By determining the logical operation order, the subsequently constructed sequence of interactive steps can have clear temporal constraints, thereby improving the consistency and interpretability of the interactive operation trajectory data in terms of logical structure.

[0150] In determining the interactive actions corresponding to at least some interface elements when state conditions are met, an interactive action refers to a specific operation type applied to an interface element to trigger a change in the interface state. For example, interactive actions may include click, double-click, long-press, swipe, drag, text input, or other touch interaction forms. Corresponding interactive actions can be generated by mapping the functional attributes and interactive features of the interface elements described in the fine-grained semantic tags. For example, when a fine-grained semantic tag indicates that an interface element is a "clickable button" and is active, a click action can be generated; when a fine-grained semantic tag indicates that an interface element is an input box and is editable, a text input action can be generated. Interactive actions may also include additional parameters, such as click coordinates, input text content, or swipe direction information. These parameters can be determined based on the coordinates of the valid element area and the fine-grained semantic tags. This embodiment does not limit the specific composition of the interactive action parameters, as long as they accurately describe the operational behavior towards the interface element. By combining the state information and functional information of the interface elements to generate corresponding interactive actions, the consistency between interactive operation trajectory data and actual interface behavior can be improved.

[0151] Logical operation order refers to the order in which interactive actions are executed, determined by state dependencies. The logical operation order can be obtained by topologically sorting the state dependency graph, ensuring that no interactive action with unmet prerequisites is executed first; alternatively, it can be sorted by dependency priority using a rule engine. This embodiment does not limit the algorithm for determining the logical operation order, as long as it ensures that the execution order of interactive actions conforms to the state dependencies.

[0152] Optionally, the determined interactive actions can be linearly arranged according to the logical operation order to generate an interactive step sequence, and each operation can be recorded in a structured form, such as including interface element identifiers, interactive action types, execution order numbers, and expected state change information. The interactive step sequence can be stored using a list structure, time series structure, or other serializable data structures. This embodiment does not limit the specific data format of the interactive step sequence, as long as it can express the sequential relationship of the interactive actions.

[0153] By determining the state dependencies between interface elements and the logical operation sequence based on these dependencies, and then combining the interactive actions that satisfy the state conditions into an interactive step sequence according to the logical operation sequence, the interactive operation trajectory data has a clear pre- and post-constraint structure and causal relationship, thereby improving the logical rigor and executability of the interactive process modeling. At the same time, the training samples explicitly include the dependency structure information and operation sequence information between interface elements, enhancing the graphical user interface agent model's ability to model the interface state change path and decision chain, and improving the agent's decision consistency and generalization ability in complex multi-step interactive scenarios.

[0154] In an example application scenario, the process of generating training data for a graphical user interface (GUI) of a "login page" is illustrated. It should be noted that this is only for explaining the technical solution of this disclosure, and the specific interface type, number of interface elements, and interaction objectives are not limited.

[0155] First, the graphical user interface (GUI) image to be processed is acquired. In this embodiment, the GUI image can be a screenshot of the login page. Further, the effective element region coordinates corresponding to each interface element in the GUI image are determined. For example, interface element detection can be performed on the "Login" button on the login page to obtain the initial element region coordinates of the button. Based on these initial element region coordinates, region segmentation processing is performed on the GUI image to obtain candidate mask images corresponding to the "Login" button. The candidate mask images are used to characterize the pixel-level region contour of the button. In some example implementations, the GUI image and the initial element region coordinates can be input into a trained prompt-driven segmentation model, such as any segmentation model, so that the prompt-driven segmentation model outputs candidate mask images with the initial element positions as prompt targets. Then, effective mask images are selected based on the matching degree between the initial element region coordinates and the candidate mask images, and the effective element region coordinates of the "Login" button are generated based on the effective mask images, thereby making the effective element region coordinates more closely match the actual display boundary of the button and reducing background blending.

[0156] After obtaining the coordinates of the effective element region, the graphical user interface image can be cropped locally based on these coordinates to obtain a local sub-image corresponding to the "Login" button. Optionally, an initial cropping region can be determined based on the coordinates of the effective element region, and then a target cropping region can be formed by expanding outwards along the periphery of the initial cropping region by a preset width ratio. The graphical user interface image can then be cropped based on this target cropping region to obtain a local sub-image containing the "Login" button and a small amount of surrounding contextual information, allowing subsequent semantic analysis to be performed within a more focused and clearer visual range.

[0157] Subsequently, fine-grained semantic labels for the "Login" button can be extracted from local sub-images. For example, the local sub-image can be input into a trained Vision-Language Model (VLM), enabling the VLM to generate multi-dimensional prompts based on the interface element features in the local sub-image. Further, the multi-dimensional prompts and the local sub-image are re-input into the VLM, allowing the VLM to output interface element feature response data based on the multi-dimensional prompts, and then combine the multi-dimensional prompts and the interface element feature response data into fine-grained semantic labels. For instance, multi-dimensional prompts might include "describe the button's color" or "determine whether the button is clickable or not," and the VLM could output response data such as "the interface element is a dark blue rectangle with white text inside, currently highlighted and active," thus enabling the fine-grained semantic labels to cover information such as color features, shape features, and interaction features.

[0158] After obtaining fine-grained semantic tags, the interaction trajectory data corresponding to the graphical user interface images can be determined based on these tags. Optionally, the target interaction result corresponding to the login page can be determined first, for example, completing the login and entering subsequent pages; then, based on the fine-grained semantic tags, the state conditions that at least some interface elements need to meet to achieve the target interaction result can be determined, such as the "Login" button needing to be active, and the account and password input boxes needing to be inputtable; based on this, an interaction step sequence corresponding to at least some interface elements that meet the state conditions can be constructed based on the fine-grained semantic tags, and this interaction step sequence can be used as the interaction trajectory data. For example, the interaction step sequence can include ordered steps such as "Enter account", "Enter password", "Confirm login button is active", and "Click login" to reflect the constraint relationship between the state of interface elements and the order of operations.

[0159] Finally, graphical user interface (GUI) training data can be generated based on GUI images, valid element region coordinates, fine-grained semantic labels, and interaction trajectory data. For example, GUI training data can associate the GUI image of a login page with the valid element region coordinates of a "login" button, and further associate it with the button's fine-grained semantic labels and interaction trajectory data oriented towards the target interaction result, thereby forming structured sample data that can be used to train a GUI agent model.

[0160] Furthermore, this embodiment also provides a graphical user interface (GUI) agent training method. This GUI training data generation method can be applied to terminal devices or servers. This embodiment is not limited to this, and the following description will use the server executing the method as an example. Figure 6The illustration schematically shows a flowchart of a graphical user interface agent training method according to some embodiments of the present disclosure. Reference Figure 6 As shown, the graphical user interface agent training method may include the following steps: Step S610: Obtain graphical user interface training data generated by the graphical user interface training data generation method. The graphical user interface training data includes a graphical user interface image, as well as the effective element region coordinates, fine-grained semantic labels, and interactive operation trajectory data corresponding to each interface element in the graphical user interface image. Step S620: Input the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interaction operation trajectory data into the graphical user interface agent model to be trained for model training, and obtain the trained graphical user interface agent model; wherein, the trained graphical user interface agent model is used to automatically generate corresponding operation decision results based on the interaction state of the interface elements in the graphical user interface image to be interacted with when receiving the graphical user interface image to be interacted with, so as to realize the automated interactive control of the graphical user interface.

[0161] The graphical user interface (GUI) image represents the overall visual content of the current interface; the coordinates of the effective element regions indicate the spatial location range of each interface element in the GUI image; fine-grained semantic labels describe the visual and interactive features of the interface elements; and the interaction trajectory data depicts the sequence of interaction steps required to achieve the target interaction result. In the specific implementation, the GUI training data can be stored in a database or file system, such as a Structured Query Language (SQL) database, a NoSQL database, or a distributed file storage system. Alternatively, it can be stored serially using JSON format files or protocol buffer formats. This embodiment does not limit the storage method of the GUI training data, as long as the association between the GUI image and the corresponding coordinates of the effective element regions, fine-grained semantic labels, and interaction trajectory data can be completely preserved.

[0162] During the acquisition process, data can be loaded according to preset data batching rules, such as reading in batches by scene type, interface category, or interaction task category, to adapt to the batch processing mechanism of model training. By acquiring graphical user interface training data with complete structure and clear information correlation, a multi-dimensional supervisory signal input basis can be provided for the graphical user interface intelligent agent model.

[0163] The graphical user interface (GUI) agent model to be trained refers to the model structure used to learn the mapping relationship between interface element understanding and interaction decisions. The GUI agent model can employ a deep neural network structure, such as extracting GUI image features based on a convolutional neural network and combining it with a Transformer architecture for cross-modal feature fusion; alternatively, it can employ a multimodal fusion model structure, jointly modeling GUI image features, effective element region coordinate encoding features, and fine-grained semantic label text embedding vectors. This embodiment does not limit the specific network structure of the GUI agent model, as long as it can receive the aforementioned input data and output operation decision results.

[0164] In the specific implementation process, visual features can first be extracted from the graphical user interface (GUI) image, and the coordinates of effective element regions can be encoded into position vectors. Fine-grained semantic labels can be converted into semantic vectors through a text encoding model, and interactive operation trajectory data can be used as supervision labels to guide the model in learning the mapping relationship from interface states to interaction step sequences. The model training process can adopt a self-supervised learning approach, updating parameters by minimizing the loss function between the predicted interaction step sequence and the interactive operation trajectory data; alternatively, a reinforcement learning approach can be used, using the interactive operation trajectory data as a reference trajectory and optimizing the decision policy through a policy gradient method. This embodiment does not limit the model training algorithm, as long as it can optimize the parameters of the GUI agent model based on the GUI training data. Through multimodal data joint training, the ability of the GUI agent model to model the relationship between interface visual features, semantic information, and operational logic can be improved.

[0165] After obtaining the trained graphical user interface (GUI) agent model, this model automatically generates corresponding operation decisions based on the interaction states of interface elements in the GUI image to be interacted with, upon receiving the GUI image to be interacted with. In practical applications, when a new GUI image is input, the GUI agent model can extract features and recognize the state of the image, combining this with the semantic representations and interaction logic representations of the interface elements already learned within the model, to output corresponding interactive actions or sequences of interactive steps as operation decisions. These operation decisions can manifest as a set of instructions for clicking a specific interface element, inputting specified content, or performing other interactive actions.

[0166] By acquiring graphical user interface (GUI) training data and training the GUI agent model, the model can automatically generate operation decision results based on the interaction state of interface elements in the GUI image, thereby improving the accuracy and generalization ability of automated interactive control of the GUI and enhancing the agent's adaptive decision-making ability in different interface scenarios.

[0167] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0168] Furthermore, in this example embodiment, a graphical user interface training data generation apparatus is also provided. (Refer to...) Figure 7 As shown, the graphical user interface training data generation device 700 includes: an effective element region determination module 710, an element local image cropping module 720, an interactive operation trajectory determination module 730, and a training data construction module 740. Wherein: The effective element region determination module 710 is used to acquire the graphical user interface image to be processed and determine the effective element region coordinates corresponding to each interface element in the graphical user interface image. The element local image cropping module 720 is used to crop the graphical user interface image locally according to the effective element region coordinates to obtain local sub-images corresponding to each of the interface elements. The interactive operation trajectory determination module 730 is used to extract fine-grained semantic tags of each interface element through the local sub-image, and determine the interactive operation trajectory data corresponding to the graphical user interface image based on the fine-grained semantic tags. The training data construction module 740 is used to generate the graphical user interface training data based on the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interactive operation trajectory data.

[0169] In some example embodiments of this disclosure, based on the foregoing scheme, the effective element region determination module 710 is configured as follows: The graphical user interface image is subjected to interface element detection to obtain the initial element region coordinates of each interface element; Based on the initial element region coordinates, the graphical user interface image is segmented to obtain candidate mask images for each interface element. Based on the degree of matching between the initial element region coordinates and the candidate mask image, a valid mask image is selected from the candidate mask images, and the valid element region coordinates are generated using the valid mask image.

[0170] In some example embodiments of this disclosure, based on the foregoing scheme, the effective element region determination module 710 is configured as follows: Calculate the degree of overlap between the initial element region coordinates and the candidate mask image; Candidate mask images with an overlap degree greater than or equal to a preset overlap degree threshold are used as valid mask images.

[0171] In some example embodiments of this disclosure, based on the foregoing scheme, the effective element region determination module 710 is configured as follows: The graphical user interface image is input into a trained point prediction multimodal model to obtain the initial element region coordinates of each interface element; the point prediction multimodal model is used to perform global detection on the graphical user interface image.

[0172] In some example embodiments of this disclosure, based on the foregoing scheme, the effective element region determination module 710 is configured as follows: The graphical user interface image and the initial element region coordinates are input into a trained cue-driven segmentation model to obtain candidate mask images for each interface element; the cue-driven segmentation model is used to perform element mask recognition on the graphical user interface image using the initial element region coordinates as cue.

[0173] In some example embodiments of this disclosure, based on the foregoing scheme, the element local image cropping module 720 is configured as follows: The initial cropping area is determined based on the coordinates of the effective element area, and the target cropping area is obtained by expanding the area outward along the periphery of the initial cropping area by a preset width ratio. The graphical user interface image is cropped according to the target cropping region to obtain local sub-images corresponding to each interface element.

[0174] In some example embodiments of this disclosure, based on the foregoing scheme, the interactive operation trajectory determination module 730 is configured as follows: Perform multi-dimensional semantic analysis on the local sub-image to generate multi-dimensional prompt questions targeting the interface element features in the local sub-image; Based on the multi-dimensional prompt questions and the local sub-images, interface element feature answer data corresponding to the local sub-images is generated; The multi-dimensional prompt questions and the interface element feature answer data are used as fine-grained semantic labels for interface elements in the local sub-image.

[0175] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the interface element features include at least one of color features, brightness features, shape features, and interaction features; The interactive features include active state, disabled state, selected state, or highlighted state.

[0176] In some example embodiments of this disclosure, based on the foregoing scheme, the interactive operation trajectory determination module 730 is configured as follows: Determine the target interaction result corresponding to the graphical user interface image; Based on the fine-grained semantic tags, determine the state conditions that at least some interface elements need to satisfy when achieving the target interaction result; Based on the fine-grained semantic tags, an interaction step sequence corresponding to at least some interface elements is constructed when the state conditions are met, and the interaction step sequence is used as the interaction operation trajectory data corresponding to the graphical user interface image.

[0177] In some example embodiments of this disclosure, based on the foregoing scheme, the interactive operation trajectory determination module 730 is configured as follows: Based on the fine-grained semantic tags, determine the state dependencies between at least some interface elements, and determine the logical operation order corresponding to at least some of the interface elements according to the state dependencies; The interactive actions corresponding to at least some of the interface elements when the state conditions are met are determined, and the interactive actions are combined according to the logical operation sequence to form the interactive step sequence.

[0178] The specific details of each module of the graphical user interface training data generation device mentioned above have been described in detail in the corresponding graphical user interface training data generation method, so they will not be repeated here.

[0179] Furthermore, in this example embodiment, a graphical user interface intelligent agent training device is also provided. (Refer to...) Figure 8 As shown, the graphical user interface intelligent agent training device 800 includes: a training data acquisition module 810 and an intelligent agent model training module 820. Wherein: The training data acquisition module 810 is used to acquire graphical user interface training data generated by the graphical user interface training data generation method. The graphical user interface training data includes a graphical user interface image, as well as the effective element region coordinates, fine-grained semantic labels and interactive operation trajectory data corresponding to each interface element in the graphical user interface image. The intelligent agent model training module 820 is used to input the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interaction operation trajectory data into the graphical user interface intelligent agent model to be trained for model training, thereby obtaining a trained graphical user interface intelligent agent model; wherein, the trained graphical user interface intelligent agent model is used to automatically generate corresponding operation decision results based on the interaction state of the interface elements in the graphical user interface image to be interacted with when receiving the graphical user interface image to be interacted with, so as to realize automated interactive control of the graphical user interface.

[0180] The specific details of each module of the graphical user interface intelligent agent training device mentioned above have been described in detail in the corresponding graphical user interface intelligent agent training method, so they will not be repeated here.

[0181] It should be noted that although several modules or units of the graphical user interface training data generation device and the graphical user interface intelligent agent training device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0182] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described graphical user interface training data generation method is also provided.

[0183] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be embodied in the following forms: a completely hardware embodiment, a completely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0184] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0185] like Figure 9 As shown, the electronic device 900 is presented in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including storage unit 920 and processing unit 910), and a display unit 940.

[0186] The storage unit stores program code that can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 910 can perform actions such as... Figure 1Step S110 shows the acquisition of a graphical user interface (GUI) image to be processed and the determination of the effective element region coordinates corresponding to each interface element in the GUI image; Step S120 shows the local cropping of the GUI image based on the effective element region coordinates to obtain local sub-images corresponding to each interface element; Step S130 shows the extraction of fine-grained semantic tags for each interface element from the local sub-images and the determination of interactive operation trajectory data corresponding to the GUI image based on the fine-grained semantic tags; Step S140 shows the generation of GUI training data based on the GUI image, the effective element region coordinates, the fine-grained semantic tags, and the interactive operation trajectory data.

[0187] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 921 and / or cache memory (Cache) 922, and may further include read-only memory (ROM) 923.

[0188] Storage unit 920 may also include a program / utility 924 having a set (at least one) program module 925, such program module 925 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0189] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0190] Electronic device 900 can also communicate with one or more external devices 970 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0191] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0192] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0193] refer to Figure 10 As shown, a program product 1000 for implementing the graphical user interface training data generation method described in embodiments of the present disclosure is illustrated. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0194] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0195] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0196] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0197] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0198] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0199] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0200] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0201] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for generating training data for a graphical user interface, characterized in that, include: Acquire the graphical user interface image to be processed, and determine the effective element region coordinates corresponding to each interface element in the graphical user interface image; The graphical user interface image is cropped locally based on the coordinates of the effective element region to obtain a local sub-image corresponding to each of the interface elements. Fine-grained semantic tags are extracted from each interface element through the local sub-image, and interactive operation trajectory data corresponding to the graphical user interface image are determined based on the fine-grained semantic tags. The graphical user interface training data is generated based on the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interactive operation trajectory data.

2. The method according to claim 1, characterized in that, Determining the effective element region coordinates corresponding to each interface element in the graphical user interface image includes: The graphical user interface image is subjected to interface element detection to obtain the initial element region coordinates of each interface element; Based on the initial element region coordinates, the graphical user interface image is segmented to obtain candidate mask images for each interface element. Based on the degree of matching between the initial element region coordinates and the candidate mask image, a valid mask image is selected from the candidate mask images, and the valid element region coordinates are generated using the valid mask image.

3. The method according to claim 2, characterized in that, The step of filtering valid mask images from the candidate mask images based on the matching degree between the initial element region coordinates and the candidate mask images includes: Calculate the degree of overlap between the initial element region coordinates and the candidate mask image; Candidate mask images with an overlap degree greater than or equal to a preset overlap degree threshold are used as valid mask images.

4. The method according to claim 2, characterized in that, The step of detecting interface elements in the graphical user interface image to obtain the initial element region coordinates of each interface element includes: The graphical user interface image is input into a trained point prediction multimodal model to obtain the initial element region coordinates of each interface element; the point prediction multimodal model is used to perform global detection on the graphical user interface image.

5. The method according to claim 2, characterized in that, The step of performing region segmentation processing on the graphical user interface image based on the initial element region coordinates to obtain candidate mask images for each interface element includes: The graphical user interface image and the initial element region coordinates are input into a trained cue-driven segmentation model to obtain candidate mask images for each interface element; the cue-driven segmentation model is used to perform element mask recognition on the graphical user interface image using the initial element region coordinates as cue.

6. The method according to claim 1, characterized in that, The step of cropping the graphical user interface image based on the coordinates of the effective element region to obtain local sub-images corresponding to each interface element includes: The initial cropping area is determined based on the coordinates of the effective element area, and the target cropping area is obtained by expanding the area outward along the periphery of the initial cropping area by a preset width ratio. The graphical user interface image is cropped according to the target cropping region to obtain local sub-images corresponding to each interface element.

7. The method according to claim 1, characterized in that, The step of extracting fine-grained semantic tags for each interface element from the local sub-image includes: Perform multi-dimensional semantic analysis on the local sub-image to generate multi-dimensional prompt questions targeting the interface element features in the local sub-image; Based on the multi-dimensional prompt questions and the local sub-images, interface element feature answer data corresponding to the local sub-images is generated; The multi-dimensional prompt questions and the interface element feature answer data are used as fine-grained semantic labels for interface elements in the local sub-image.

8. The method according to claim 7, characterized in that, The interface element features include at least one of color features, brightness features, shape features, and interaction features; The interactive features include active state, disabled state, selected state, or highlighted state.

9. The method according to claim 1, characterized in that, The step of determining the interactive operation trajectory data corresponding to the graphical user interface image based on the fine-grained semantic tags includes: Determine the target interaction result corresponding to the graphical user interface image; Based on the fine-grained semantic tags, determine the state conditions that at least some interface elements need to satisfy when achieving the target interaction result; Based on the fine-grained semantic tags, an interaction step sequence corresponding to at least some interface elements is constructed when the state conditions are met, and the interaction step sequence is used as the interaction operation trajectory data corresponding to the graphical user interface image.

10. The method according to claim 9, characterized in that, The step sequence for constructing the interaction steps corresponding to at least some interface elements when satisfying the state conditions based on the fine-grained semantic tags includes: Based on the fine-grained semantic tags, determine the state dependencies between at least some interface elements, and determine the logical operation order corresponding to at least some of the interface elements according to the state dependencies; The interactive actions corresponding to at least some of the interface elements when the state conditions are met are determined, and the interactive actions are combined according to the logical operation sequence to form the interactive step sequence.

11. A graphical user interface intelligent agent training method, characterized in that, include: Obtain graphical user interface training data generated by the graphical user interface training data generation method according to any one of claims 1-10, wherein the graphical user interface training data includes a graphical user interface image, and effective element region coordinates, fine-grained semantic labels and interactive operation trajectory data corresponding to each interface element in the graphical user interface image. The graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interaction operation trajectory data are input into the graphical user interface agent model to be trained for model training, and a trained graphical user interface agent model is obtained. The trained graphical user interface (GUI) agent model is used to automatically generate corresponding operation decision results based on the interaction state of interface elements in the GUI image to be interactively controlled when a GUI image to be interactively controlled is received, so as to realize automated interactive control of the GUI.

12. A graphical user interface training data generation device, characterized in that, include: The effective element region determination module is used to acquire the graphical user interface image to be processed and determine the coordinates of the effective element region corresponding to each interface element in the graphical user interface image. The element local image cropping module is used to crop the graphical user interface image locally according to the coordinates of the effective element region to obtain local sub-images corresponding to each of the interface elements. The interactive operation trajectory determination module is used to extract fine-grained semantic tags of each interface element through the local sub-image, and determine the interactive operation trajectory data corresponding to the graphical user interface image based on the fine-grained semantic tags. The training data construction module is used to generate the graphical user interface training data based on the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels, and the interactive operation trajectory data.

13. A graphical user interface intelligent agent training device, characterized in that, include: The training data acquisition module is used to acquire graphical user interface training data generated by the graphical user interface training data generation method according to any one of claims 1-10. The graphical user interface training data includes a graphical user interface image, as well as the effective element region coordinates, fine-grained semantic labels and interactive operation trajectory data corresponding to each interface element in the graphical user interface image. The intelligent agent model training module is used to input the graphical user interface image, the coordinates of the effective element region, the fine-grained semantic labels and the interaction operation trajectory data into the graphical user interface intelligent agent model to be trained, so as to obtain the trained graphical user interface intelligent agent model. The trained graphical user interface (GUI) agent model is used to automatically generate corresponding operation decision results based on the interaction state of interface elements in the GUI image to be interactively controlled when a GUI image to be interactively controlled is received, so as to realize automated interactive control of the GUI.

14. An electronic device, characterized in that, include: processor; as well as The memory stores computer-readable instructions that, when executed by the processor, implement the graphical user interface training data generation method as described in any one of claims 1-10, or the graphical user interface agent training method as described in claim 11.

15. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the graphical user interface training data generation method as described in any one of claims 1-10, or the graphical user interface intelligent agent training method as described in claim 11.

Images