A graphical user interface (GUI) data generation method and apparatus, an electronic device, and a storage medium

Abstract

The present disclosure relates to a graphical user interface (GUI) data generation method and device, an electronic device and a storage medium. The method comprises: performing interactive element detection on a GUI image to obtain position information of a target interactive element; performing image analysis on the target interactive element in the GUI image based on the position information of the target interactive element to generate a description text; the description text is used to describe visual information and functional information of the target interactive element; and the position information of the target interactive element and the description text are integrated to obtain GUI data. The embodiments of the present disclosure can output multi-modal GUI data containing visual information and functional information, and improve the generalization ability and practicability of a trained GUI understanding model in actual interaction.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology, and particularly relates to a graphical user interface (GUI) data generation method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the development of computer vision and automation technology, automatic generation and annotation of graphical user interface (GUI) data has become a key requirement for improving human-computer interaction intelligence. For example, in scenarios such as intelligent assistants, automated testing, and accessibility services, the system needs to accurately understand the visual features and functional semantics of interface elements to execute user instructions (such as "click the blue circle save button" or "enter the keyword in the search box"). However, the GUI data generated by traditional methods is difficult to meet the needs of multi-modal information in complex interaction scenarios.

[0003] In related technologies, GUI data is mainly generated in two ways: one is to extract element coordinates and type labels based on structured data (such as HTML, Accessibility Trees); the other is to rely on manual annotation or synthetic tools to generate simple annotations. However, the GUI data obtained in related technologies is limited to the bounding box coordinates and basic classification labels of elements, such as "button" and "input box".

[0004] Such coarse-grained GUI data makes it difficult for the trained GUI understanding model to parse user natural language instructions, severely limiting the model's generalization ability and practicality in actual interactions. SUMMARY

[0005] In view of this, the present disclosure provides a technical solution for GUI data generation.

[0006] According to an aspect of the present disclosure, a graphical user interface (GUI) data generation method is provided, comprising:

[0007] detecting interactive elements in a GUI image to obtain position information of a target interactive element;

[0008] performing image analysis on the target interactive element in the GUI image based on the position information of the target interactive element to generate a description text; the description text is used to describe visual information and functional information of the target interactive element;

[0009] integrating the position information of the target interactive element with the description text to obtain GUI data.

[0010] In one possible implementation, the detecting interactive elements in the GUI image to obtain position information of a target interactive element comprises:

[0011] identifying non-text interactive elements and text elements in the GUI image;

[0012] According to the spatial relationship and the overlapping degree between the non-text interactive elements and the text elements, redundant elements are removed to obtain position information of the target interactive element.

[0013] In a possible implementation, the removing of the redundant elements according to the spatial relationship and the overlapping degree between the non-text interactive elements and the text elements to obtain the position information of the target interactive element includes:

[0014] In a case where the detection box of the text element is contained in the detection box of the non-text interactive element, the text element is determined as a redundant element and is removed;

[0015] In a case where the detection box of the text element is completely located outside the detection box of the non-text interactive element, the text element is determined as a non-interactive element and is removed.

[0016] In a possible implementation, the removing of the redundant elements according to the spatial relationship and the overlapping degree between the non-text interactive elements and the text elements to obtain the position information of the target interactive element includes:

[0017] The overlapping proportion of the detection boxes of the text element and the non-text interactive element is calculated, the overlapping proportion being a ratio of an intersection area of the two detection boxes to a union area of the two detection boxes;

[0018] In a case where the overlapping proportion is lower than a preset threshold, the text element is determined as a non-interactive element and is removed.

[0019] In a possible implementation, the detecting of the GUI image to obtain the position information of the target interactive element includes:

[0020] The detected interactive elements are sampled in spatial distribution, and an interactive element with discrete distribution is selected as the target interactive element.

[0021] In a possible implementation, the sampling of the detected interactive elements in spatial distribution to select an interactive element with discrete distribution as the target interactive element includes:

[0022] An initial interactive element is randomly selected from the detected interactive elements;

[0023] The distance between a remaining unselected element and a selected element selected last time is calculated, and an unselected element is randomly selected from the first N unselected elements with the largest distance; wherein N is a positive integer.

[0024] The calculation is iteratively performed until a preset sampling number of selected elements is reached.

[0025] In a possible implementation, the image analysis of the target interactive element in the GUI image based on the position information of the target interactive element comprises:

[0026] The visual analysis and semantic analysis of the target interactive element in the GUI image based on the position information of the target interactive element comprises:

[0027] The description text is generated based on the visual information and the function information.

[0028] In a possible implementation, the image analysis of the target interactive element in the GUI image based on the position information of the target interactive element comprises:

[0029] The GUI image containing the position information of the target interactive element and the prompt text are input into a large-scale visual language model (LVLM) to obtain the description text of the target interactive element; the description text is a text describing the target interactive element based on natural language.

[0030] In a possible implementation, the GUI image is a screenshot of a cross-platform desktop application.

[0031] According to another aspect of the present disclosure, a graphical user interface (GUI) data generation apparatus is provided, comprising:

[0032] A detection module is configured to detect interactive elements in a GUI image to obtain position information of a target interactive element.

[0033] A description text generation module is configured to perform image analysis of the target interactive element in the GUI image based on the position information of the target interactive element to generate a description text; the description text is used to describe visual information and function information of the target interactive element.

[0034] An integration module is configured to integrate the position information of the target interactive element and the description text to obtain GUI data.

[0035] In a possible implementation, the detection module is configured to:

[0036] identify non-text interactive elements and text elements in the GUI image;

[0037] According to spatial relationships and overlapping degrees between the non-text interactive elements and the text elements, redundant elements are removed to obtain the position information of the target interactive element.

[0038] In a possible implementation, the detection module is configured to:

[0039] In a case where the detection box of the text element is contained in the detection box of the non-text interactive element, the text element is determined as a redundant element and is removed.

[0040] In a case where the detection box of the text element is completely located outside the detection box of the non-text interactive element, the text element is determined as a non-interactive element and is removed.

[0041] In a possible implementation, the detection module is configured to:

[0042] The detection module is configured to calculate an overlap ratio of the detection boxes of the text element and the non-text interactive element, the overlap ratio being a ratio of an intersection area to a union area of the two detection boxes.

[0043] In a case where the overlap ratio is lower than a preset threshold, the text element is determined as a non-interactive element and is removed.

[0044] In a possible implementation, the detection module is configured to:

[0045] The detection module is configured to perform spatial distribution sampling on the detected interactive elements, and select interactive elements with a distributed discrete as target interactive elements.

[0046] In a possible implementation, the detection module is configured to:

[0047] The detection module is configured to randomly select an initial interactive element from the detected interactive elements.

[0048] The detection module is configured to calculate distances between remaining unselected elements and a selected element selected last time, and randomly select an unselected element from a first N unselected elements with the largest distances, where N is a positive integer.

[0049] The detection module is configured to repeat the process of calculating and selecting an unselected element until a preset sampling number of selected elements is reached.

[0050] In a possible implementation, the description text generation module is configured to

[0051] The description text generation module is configured to perform visual analysis and semantic analysis on the target interactive element in the GUI image based on position information of the target interactive element, to obtain visual information and function information of the target interactive element.

[0052] The description text generation module is configured to generate a description text based on the visual information and the function information.

[0053] In a possible implementation, the description text generation module is configured to

[0054] The GUI image containing the position information of the target interactive element and prompt text are input into a large-scale visual language model LVLM to obtain description text of the target interactive element; the description text is text describing the target interactive element based on natural language.

[0055] In a possible implementation, the GUI image is a screenshot of a cross-platform desktop application.

[0056] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the above method.

[0057] According to another aspect of the present disclosure, a non-volatile computer readable storage medium is provided, having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the above method.

[0058] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, or a non-volatile computer readable storage medium carrying the computer program, the computer program being executed by a processor to implement the steps of the above method.

[0059] In the embodiments of the present disclosure, the position information of the target interactive element is obtained by interactive element detection on the GUI image; the target interactive element in the GUI image is analyzed based on the position information of the target interactive element to generate description text; the description text is used to describe the visual information and functional information of the target interactive element; and the position information of the target interactive element and the description text are integrated to obtain GUI data. Thus, the obtained description text breaks through the limitation of traditional GUI data that only records coordinates or simple labels, and through the combination of visual analysis and semantic analysis, multi-modal description containing color, shape and functional information is output, so that the information density of the GUI data is significantly improved, the GUI understanding model training that needs to understand complex instructions of multi-modal is directly supported, the generalization ability and practicality of the trained GUI understanding model in actual interaction are improved, in addition, the description text is obtained by image analysis on the GUI image, so that the GUI image can be not limited by the platform, and the generated GUI data provides a high-quality data basis for cross-platform and multi-scene GUI understanding tasks.

[0060] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0061] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0062] Figure 1 A flowchart illustrating a GUI data generation method according to an embodiment of the disclosure is shown.

[0063] Figure 2 A flowchart illustrating another GUI data generation method according to an embodiment of the disclosure is shown.

[0064] Figure 3 A block diagram illustrating a graphical user interface (GUI) data generation apparatus according to an embodiment of the disclosure is shown.

[0065] Figure 4 is a block diagram of an apparatus 1900 for graphical user interface (GUI) data generation according to an example embodiment. DETAILED DESCRIPTION

[0066] Various example embodiments, features and aspects of the disclosure will be explained below in detail with reference to the accompanying drawings. The same reference numerals are used throughout the drawings and same or similar functions are referred to by the same or similar reference numerals to prevent obscuring the subject matter of the present disclosure. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale nor in proportion.

[0067] As used herein, the terms "include," "comprise," "have," or their variants are open-ended, and include one or more stated features, integers, elements, steps, components or functions but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof.

[0068] When an element is referred to as being "connected," "coupled," "responsive," or "related" to another element, it can be directly connected, coupled, responsive, or related to the other element, or intervening elements can be present.

[0069] Although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments can be termed a second element / operation in other embodiments without departing from the teachings of the present inventive concept.

[0070] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0071] In addition, for better illustration of the present disclosure, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, elements and circuits well known to those skilled in the art are not described in detail in order to highlight the main idea of the present disclosure.

[0072] As described above, the GUI data obtained in the related art is limited to the bounding box coordinates of the elements and the basic classification labels such as "button" and "input box". Based on this, the present disclosure proposes a GUI data generation method, and the technical solution provided by the present disclosure exhibits important value in a plurality of practical application scenarios.

[0073] For example, in the field of intelligent interactive assistants, users often operate graphical interfaces through natural language instructions, such as "click the blue circle in the upper left corner to save the button". Traditional systems are difficult to accurately understand such instructions due to the lack of detailed descriptions of the visual attributes and functional semantics of the elements. In the present disclosure, however, the description text containing the visual information and functional information of the target interactive element is automatically generated, so that the intelligent assistant can accurately locate the target interactive element required by the user according to the description text, significantly improving the interaction success rate.

[0074] Similarly, in automated testing tools, test scripts need to verify whether the functions of interface elements meet expectations (such as "check whether the search box supports paste operations"). Traditional methods rely on simple labels and coordinates and cannot describe the interactive characteristics of elements, while the multi-modal data provided by the present solution can directly guide the test scripts to simulate real user behavior, enhancing test coverage and accuracy.

[0075] In addition, in the context of accessibility services, visually impaired people rely on voice descriptions to interact with interfaces. Traditional annotation data only provides "button-coordinate" information, while the descriptions such as "red warning icon, indicating network connection failure" generated by the present solution can be converted into more natural voice feedback, significantly improving the usability of accessibility services.

[0076] The GUI data generation method provided by the present disclosure does not depend on the GUI screenshots of specific operating systems, and therefore can be applied to application scenarios such as intelligent interaction, automated testing and accessibility services in different operating systems.

[0077] Figure 1 A flowchart of a GUI data generation method according to an embodiment of the present disclosure is shown. As shown in Figure 1 The method includes:

[0078] In step S11, the GUI image is subjected to interactive element detection to obtain the position information of the target interactive element;

[0079] The GUI image is a visual presentation form of a graphical user interface in a computer system, usually in the form of an image file or a screenshot, which can be a screenshot of a desktop application, a web interface or a mobile application, and covers visual elements of user interaction with the system.

[0080] In a possible implementation, the GUI image is a screenshot of a cross-platform desktop application.

[0081] The platform in the present disclosure is used to provide a running environment (resources) for an application program. Different platforms have different interface layouts, and the interface layouts of application programs in different platforms are also different, which leads to different distributions of interactive elements in GUIs.

[0082] The GUI image processed by the present disclosure is not limited to a specific platform. In related technologies, GUI data is generated mainly depending on screenshots or synthetic data of a specific platform (such as Windows), which leads to uneven distribution of cross-platform data. For example, the traditional method can only generate annotation data for Windows desktop applications, and the interface characteristics (such as menu bar layout and control style) of macOS or Linux platforms are not fully covered.

[0083] The GUI data generation method of the present disclosure can adapt to graphical interfaces of different platforms, such as GUI images in Windows, macOS, Linux and other operating system platforms, without relying on the characteristics or limitations of a specific platform.

[0084] An interactive element is a visual component in an interface that can be operated or interacted with by a user. The interactive element can include at least one of the following: buttons, input boxes, icons, menu bars, and clickable or editable text areas. For example, a “save” button, a “search” input box, a Wi-Fi connection icon, a checkbox in a file list, and the like.

[0085] When detecting interactive elements in a GUI image, image recognition technology can be used to identify operable components in the GUI image and determine their positions to adapt to GUI images collected by different platforms. For example, a visual detection model can be used to identify icons, controls and other non-text interactive elements, a text recognition technology can be used to identify text content as a text element, and the position information of the identified elements can be located. The text element can be an interactive element or a non-interactive element, and the non-interactive text element will be further removed in the subsequent process.

[0086] For example, a detection module in the visual detection model OmniParser can be used to detect folder icons and application shortcut icons in a desktop screenshot, and a text recognition tool PaddleOCR can be used to recognize text in the interface and output the position information of the recognized elements.

[0087] The position information of the target interactive element is used to describe the spatial coordinates or the area range of the target interactive element in the interface, which can be expressed in the form of the coordinates of a bounding box, such as the pixel coordinates of the upper left corner and the lower right corner. For example, the position information of a rectangular button can be represented as (x1=100, y1=200, x2=150, y2=250).

[0088] In step S12, the target interactive element in the GUI image is analyzed based on the position information of the target interactive element to generate a description text; the description text is used to describe the visual information and the functional information of the target interactive element.

[0089] The description text is used to linguistically describe the attributes and functions of the interactive element, and can include text content of visual features and functional semantics. The visual features include color, shape, size, and the like, and the functional semantics include use, operation instruction, and the like. For example, “blue circular save button”, “white rectangular input box with ‘search’ label”.

[0090] In generating the description text, the visual and functional attributes of the corresponding position can be analyzed based on the position information of the target interactive element in the GUI image, and then converted into a language description. Specifically, the feature extraction and semantic inference can be performed on the image region where the target interactive element is located, and based on the extracted visual information such as color and shape, and the semantic inference of the functional use, a multi-modal description of the target interactive element is generated.

[0091] For example, after detecting a blue circular icon, the description text “blue circular save button” is generated in combination with the text “save” at the position of the icon.

[0092] In step S13, the position information of the target interactive element is integrated with the description text to obtain GUI data.

[0093] In the integration, the position information and the description text can be bound according to a preset rule to form a structured data entry. For example, the coordinates (100, 200, 150, 250) of the button and the description “blue circular save button” are combined into one GUI data record.

[0094] In one possible implementation, the integration of the position information of the target interactive element and the description text includes: the position information of the interactive element and the description text are associatedly stored to generate GUI data in a standard format.

[0095] GUI data, used to describe interface content, provides standardized input for machine processing and model training by structuring the visual attributes, functional semantics, and spatial positions of interface elements. For example, a piece of GUI data can contain the coordinate range of a button, color code (such as "#007BFF"), shape description (such as "circle"), and function label (such as "save"). This structured format facilitates direct parsing by machine learning algorithms, allowing features to be extracted without additional preprocessing.

[0096] GUI data can be directly used to train GUI understanding models, which are multi-modal models based on visual information and textual information. In subsequent multi-modal model training based on GUI data, the GUI data generated in the embodiments of the present disclosure can serve multiple modal tasks at the same time. Specifically, the visual model in the multi-modal model can learn element positioning using the coordinate and shape information in the multi-modal GUI data, the natural language processing model can understand user instructions through functional description, and the end-to-end GUI understanding model can be jointly trained to achieve precise execution of instructions such as "click the blue circular save button".

[0097] In addition, the rich semantics of the GUI data generated in the embodiments of the present disclosure can improve the generalization ability of the multi-modal model obtained by training. Traditional coordinate-label data only supports basic classification tasks, while annotation data containing color, shape, and function description enables the multi-modal model to associate visual features with semantic intent. For example, when training an automated testing model based on a multi-modal model, the system can not only recognize the existence of a "button", but also verify whether its function meets the design expectation according to descriptions such as "blue" and "save". This multi-dimensional information integration enables the model to maintain high accuracy when facing different operating system platforms or interface design styles, significantly reducing the cost of cross-scene adaptation.

[0098] In the embodiments of the present disclosure, the position information of the target interactive element is obtained by interactive element detection on the GUI image; the image analysis is performed on the target interactive element in the GUI image based on the position information of the target interactive element, and the description text is generated; the description text is used to describe the visual information and the functional information of the target interactive element; and the position information of the target interactive element is integrated with the description text to obtain the GUI data. Thus, the description text obtained breaks through the limitation of the traditional GUI data that only records coordinates or simple labels, and through the combination of visual analysis and semantic analysis, the multi-modal description containing color, shape and functional information is output, so that the information density of the GUI data is significantly improved, the GUI understanding model training that needs to understand multi-modal complex instructions is directly supported, the generalization ability and practicality of the trained GUI understanding model in actual interaction are improved, and in addition, the description text is obtained by image analysis on the GUI image, so that the GUI image can not be limited by the platform, and the generated GUI data provides a high-quality data basis for cross-platform and multi-scene GUI understanding tasks.

[0099] In a possible implementation, the interactive element detection on the GUI image to obtain the position information of the target interactive element includes: identifying non-text interactive elements and text elements in the GUI image; and according to the spatial relationship and the overlapping degree between the non-text interactive elements and the text elements, eliminating redundant elements to obtain the position information of the target interactive element.

[0100] The redundant elements can be interface elements that are incorrectly identified or repeatedly labeled in the detection process, for example, the detection boxes of the same control and the detection boxes of the corresponding text, non-interactive decorative text, and the like.

[0101] The valid interactive elements can be screened by distinguishing the non-text interactive elements and the text elements and combining the spatial relationship thereof. The spatial relationship is used to describe the relative position and layout of the elements in the interface, including geometric features such as overlapping area, distance, and containment relationship. The non-text interactive elements include icons, buttons and the like, and the text elements include words.

[0102] In the related art, the detection of the non-text interactive elements and the text elements is usually performed independently, so that the correlation between the two is ignored. For example, a “confirm” button can include text and icons, and if the two are detected as two independent elements, redundant elements are generated.

[0103] Then, by respectively identifying the non-text interactive elements and the text elements and judging whether they belong to the same interactive unit based on the spatial relationship (such as the overlapping degree and the relative position) thereof, the target interactive element that really needs to be obtained can be more accurately positioned, the redundant elements that interfere are eliminated, and the reliability of the detection result is improved.

[0104] In a possible implementation, the position information of the target interactive element is obtained by removing redundant elements according to the spatial relationship and overlapping degree between the non-text interactive element and the text element, including: in a case where the detection box of the text element is contained in the detection box of the non-text interactive element, determining that the text element is a redundant element and removing it; in a case where the detection box of the text element is completely located outside the detection box of the non-text interactive element, determining that the text element is a non-interactive element and removing it.

[0105] In this implementation, the elements that need to be retained or removed can be determined by the spatial containment relationship between the detection boxes of the text element and the non-text interactive element. The detection box is a rectangular region surrounding the interface element detected, and the position and range of the element are represented in the form of coordinates (such as the upper left corner and the lower right corner).

[0106] The detection box coordinates of the non-text interactive element and the text element can be obtained at the same time. For the text element, the following two types of filtering are performed according to the position relationship between its detection box and the detection box of the non-text interactive element.

[0107] If the detection box of the text is completely located in the detection box of a non-text interactive element, for example, the "save" text is embedded in the button icon, it is determined that the text is a redundant element, that is, the text is part of the icon, the text element needs to be removed, and only the detection box of the non-text interactive element is retained as the target interactive element.

[0108] If the detection box of the text element is completely located outside the detection boxes of all non-text interactive elements, for example, the copyright declaration text in the corner of the interface, it is determined that the text is a non-interactive element and needs to be removed directly.

[0109] In the embodiments of the present disclosure, in a case where the detection box of the text element is contained in the detection box of the non-text interactive element, it is determined that the text element is a redundant element and is removed; in a case where the detection box of the text element is completely located outside the detection box of the non-text interactive element, it is determined that the text element is a non-interactive element and is removed. Thus, by using the filtering rule of the spatial containment relationship, the judgment logic of the redundant element is simplified, and the detection result is ensured to be consistent with the actual interactive intention. Figure 1 At the same time, accurate filtering of redundant and invalid elements is realized, which provides a reliable guarantee for generating high-quality and low-noise GUI data.

[0110] The traditional method can label the embedded text and the control as independent elements respectively, resulting in data redundancy. For example, a button with the word "Confirm" can be split into two records of "button-coordinate" and "text-coordinate", while the user actually interacts with the button as a whole. The present scheme removes the completely contained text and retains the non-text interactive element as a representative, so that the GUI data is closer to the real operation logic and reduces the confusion of downstream model training.

[0111] In addition, by removing the text completely located outside the detection box of the non-text interactive element, the interference of non-interactive elements is reduced. For example, static text such as copyright declaration and help prompt in the interface usually does not need to be operated, and if not filtered, it can be mislabeled as an interactive element, resulting in model error learning. Through this filtering rule, such noise data can be automatically excluded, improving the purity of GUI data. For example, when training an automated testing model, the script will focus on operating the real controls (such as buttons and input boxes), rather than mistakenly clicking irrelevant text areas.

[0112] In a possible implementation, the position information of the target interactive element is obtained by removing the redundant elements according to the spatial relationship and the overlapping degree between the non-text interactive element and the text element, comprising: calculating the overlapping proportion of the detection boxes of the text element and the non-text interactive element, the overlapping proportion being the ratio of the intersection area to the union area of the two detection boxes; in the case that the overlapping proportion is lower than a preset threshold, determining that the text element is a non-interactive element and removing it.

[0113] In the case that the detection boxes of the text element and the non-text interactive element overlap, the overlapping proportion of the detection box of the text element and the detection box of the non-text interactive element can be calculated, that is, the ratio of the intersection area to the union area. Then, based on a preset threshold, the strength of the association between the text element and the non-text interactive element is determined. The preset threshold can be set according to experience or experiment, and is used to determine the strength of the association between the elements.

[0114] In the case that the overlapping proportion is lower than the preset threshold, it is determined that the text element is a non-interactive element and is removed; in the case that the overlapping proportion is not lower than the preset threshold, it is determined that the text element and the non-text interactive element are strongly associated and are interactive elements, and do not need to be removed. For example, the preset threshold can be 0.7, when the overlapping proportion is greater than or equal to 0.7, it is considered that the text element and the non-text interactive element are strongly associated, and when it is lower than 0.7, it is considered that they are not associated.

[0115] For example, if a text label partially overlaps with a button icon, the overlap ratio is 0.6, and the preset threshold is 0.7, it is determined that the text is not associated with the button, and the text is a non-interactive element (such as background description text) and needs to be removed. On the contrary, if the overlap ratio exceeds the threshold, the text element is retained or merged with the associated control.

[0116] The preset threshold can be adjusted according to actual needs to control the strictness of the screening. For example, in a high-density interface, the threshold can be reduced to retain more potential interactive text.

[0117] In the embodiments of the present disclosure, the overlap ratio of the bounding boxes of the text element and the non-text interactive element is calculated. If the overlap ratio is lower than a preset threshold, the text element is determined as a non-interactive element and is removed. In this way, the spatial overlap between the text element and the non-text interactive element is quantified to screen the effective interactive element, so as to determine whether the text is associated with the interactive control, thereby excluding irrelevant or lowly associated text elements, and enabling the detection process to adapt to complex scenarios of different interface layouts.

[0118] In addition, the numerical spatial relationship analysis and the configurable threshold mechanism significantly improve the robustness of interactive element detection, providing a technical guarantee for generating high-precision and low-noise GUI data. For example, when training a voice assistant model, the system can accurately associate the "magnifying glass icon with the'search' label", rather than mistakenly including irrelevant text in the operation range, thereby improving the success rate of user instruction execution.

[0119] In a possible implementation, the GUI image is subjected to interactive element detection to obtain position information of a target interactive element, including: performing spatial distribution sampling on the detected interactive elements, and selecting the interactive elements with discrete distribution as the target interactive elements.

[0120] Spatial distribution sampling is to select a subset with spatial representation according to the position distribution characteristics of the elements in the interface, so as to ensure the data diversity of the target interactive elements. The discrete distribution indicates the state of the target interactive elements arranged dispersedly in the interface, which is opposite to "dense concentration", and ensures the universality of the spatial coverage of the target interactive elements.

[0121] That is, after detecting a plurality of interactive elements, the interactive elements with discrete distribution in the interface are preferentially selected, so as to avoid excessive concentration of the elements in a local area, improve the diversity and representativeness of the GUI data, ensure that the generated training data can cover different functional areas of the interface, and thus enhance the understanding ability of the downstream model to the global layout.

[0122] Specifically, all interactive elements in the interface are first detected and their position information is obtained. Then, based on a spatial distribution sampling strategy, a subset of interactive elements that are far apart from each other and are distributed more dispersedly is selected as target interactive elements.

[0123] In the embodiments of the present disclosure, by performing spatial distribution sampling on the detected interactive elements, the interactive elements that are distributed dispersedly are selected as target interactive elements, which significantly improves the ability to capture global features of the interface. By preferentially selecting elements that are distributed dispersedly, the collected GUI data covers different functional blocks of the interface, such as samples that contain top menus, side bar controls, and bottom operation buttons at the same time.

[0124] In addition, selecting interactive elements that are distributed dispersedly as target interactive elements can reduce the risk of overfitting in subsequent model training. For example, if the proportion of elements in a certain area in the training data is too high (such as a frequently appearing right input box), the model may not have sufficient response capability for elements on the left or top. By spatially balanced sampling, the model can more comprehensively learn the features of elements in different positions, thereby more accurately locating the target when facing a new interface. For example, in an automated testing scenario, the script can effectively operate controls scattered throughout the interface, rather than being limited to a specific area.

[0125] Furthermore, through logical optimization of spatial distribution, the diversity and balance of GUI data are ensured, which provides key support for training a robust GUI understanding model. For example, in cross-platform applications, the layout style of different operating system platforms (such as centralized menus for Windows and decentralized designs for macOS) can be adapted, which improves the success rate of executing interaction instructions.

[0126] In a possible implementation, the spatial distribution sampling on the detected interactive elements and the selection of interactive elements that are distributed dispersedly as target interactive elements include: randomly selecting an initial interactive element from the detected interactive elements; calculating the distance between the remaining unselected elements and the selected element in the last iteration, and randomly selecting an unselected element from the first N unselected elements with the largest distance; wherein N is a positive integer; and repeating the process of calculating and selecting an unselected element until the number of selected elements reaches a preset sampling number.

[0127] When performing spatial distribution sampling on the detected interactive elements, one of all the detected interactive elements can be randomly selected as an initial element. For example, one button located at the upper left corner of the interface is randomly selected from 20 elements.

[0128] Then the post-iteration process begins, in each iteration, the distance between the remaining unselected elements and the last selected element is calculated, such as Euclidean distance or the distance between the center points of the bounding boxes. For example, if the last selected element is the button in the top left corner, the distance between the other elements and the button is calculated. Then one is randomly selected from the top N unselected elements with the largest distance. For example, if N = 3, the top 3 candidate elements with the largest distance are selected, such as the icon in the bottom right corner, the input box in the middle, and the menu item on the right, and one is randomly selected, such as the menu item on the right.

[0129] The above iteration process is repeated until the number of selected elements reaches the preset sampling number, and the preset sampling number is a target interactive element number preset according to actual needs, for example, 10 elements are sampled on each interface to balance the data volume and diversity.

[0130] In the embodiments of the present disclosure, by dynamically adjusting the reference point and the random selection mechanism, the limitations of the traditional fixed sampling strategy are solved. Each iteration takes the latest selected element as the reference, forcing the sampling to extend in different directions, thereby forming a multi-center diffusion distribution pattern. In addition, the random selection mechanism of the top N candidates enhances the adaptability of the algorithm. In an interface with uneven element distribution, if only the element with the largest distance is selected, the sampling may deviate from the core function area due to isolated elements at the edge of the interface. However, by selecting from multiple long-distance candidates, the dispersion and function coverage can be considered. Finally, through dynamic iteration and randomization balance, GUI data with extensive distribution and diversity is generated. For example, when training a cross-platform GUI understanding model, the model can recognize the layout rules in different operating system platforms (such as centralized menus in Windows and decentralized controls in macOS), and improve the generalization ability and robustness of the model in real scenarios.

[0131] In addition, the selected target interactive elements can also be sorted according to the selection order, and the target interactive elements can be labeled in order. For example, after selecting the target interactive element in each iteration, a unique identifier (such as 0, 1, 2) is assigned to each element, and it is bound with the position information and description text of the element to ensure the traceability and scalability of the data structure.

[0132] In a possible implementation, the image analysis of the target interactive element in the GUI image based on the position information of the target interactive element to generate the description text comprises: visual analysis and semantic analysis of the target interactive element in the GUI image based on the position information of the target interactive element to obtain visual information and function information of the target interactive element; and generating the description text based on the visual information and the function information.

[0133] Visual analysis is used to identify the appearance attributes of elements in GUI images, such as color, shape, texture, etc. Semantic analysis is used to analyze the position of elements in the interface, adjacent text, and type, and deduce their functional use.

[0134] Then, for the selected target interactive element, a corresponding region can be cropped from the GUI image based on the position information of the target interactive element, such as the bounding box coordinates, and visual analysis is performed on the region. Specifically, visual attributes can be extracted using image processing techniques. For example, the dominant color of an element can be determined through color recognition, such as red, blue, its geometric shape can be determined through shape detection, such as a circle, a rectangle, or the detailed features of an icon can be identified through texture analysis, such as stripes, gradients.

[0135] During semantic analysis, the type of the element (such as a button, an input box) and its surrounding context information (such as adjacent text, interface layout) can be combined to deduce its functional use. For example, if a rectangular region is detected and the adjacent text is "search", it is analyzed as a "search input box"; if a circular icon is located at the top of the toolbar and adjacent to the "settings" label, it is deduced as a "settings function button".

[0136] Then, description text is generated based on the visual information and functional information.

[0137] For example, in the interface of a cloud storage application, an interactive element with bounding box coordinates (x = 200, y = 300, width = 50, height = 50) is detected through visual analysis, its color is identified as blue through image analysis, its shape is identified as a circle, and it has a cloud pattern on its surface; then semantic analysis is performed, and according to the element being located in the "upload" function area and the adjacent text being "click to upload files", it is deduced that its function is to trigger file upload; finally, the visual and functional information is integrated to generate the description text "blue circular cloud icon for uploading files".

[0138] In the embodiments of the present disclosure, through the deep integration of visual and semantic information, the expression depth and practicality of GUI data are significantly improved. The "button-coordinate" type data generated by traditional methods can only support basic element positioning tasks, while the "blue circular upload button" type description text generated by the present solution enables the model to directly associate visual features with functional semantics. For example, in a voice interaction scenario, the user instruction "click the blue cloud icon to upload files" can be accurately mapped to the target interactive element without relying on coordinate conversion or additional context analysis.

[0139] In addition, the introduction of multi-modal descriptions enhances the model's cross-scene adaptability. In interfaces of different platforms or design styles, elements with the same function may present different visual forms (such as the flat buttons of Windows and the skeuomorphic icons of macOS). By including descriptions of properties such as color and shape, the model can identify functional consistency and reduce misjudgments caused by appearance differences. For example, if both a "blue rectangular save button" and a "gray circular save button" are included in the training data, the model can learn to ignore the visual differences and focus on the functional semantics, thereby improving the robustness of cross-platform applications.

[0140] In one possible implementation, the image analysis of the target interactive element in the GUI image based on the position information of the target interactive element generates a description text, including: inputting the GUI image containing the position information of the target interactive element and the prompt text into a large-scale visual language model (LVLM) to obtain a description text of the target interactive element; and the description text is a text describing the target interactive element based on natural language.

[0141] A large-scale visual language model (LVLM) is a multi-modal artificial intelligence model that can process images and text simultaneously, and learns the correlation between vision and language through pre-training. A prompt text is a structured instruction used to guide the LVLM to generate specific content, such as "describe the color and function of the marked element in the image".

[0142] Specifically, the GUI image and the bounding box coordinates of the target interactive element can be integrated as input data, and then a prompt text is input. For example, a bounding box of the target interactive element can be marked and labeled in the GUI image, and then a prompt text containing the label can be generated, such as "describe the UI elements in the area labeled 0-5 in the image".

[0143] Then, the image with the mark and the prompt text are input into the LVLM, and the LVLM generates a natural language description containing visual attributes and functional semantics based on the understanding of the image content and the guidance of the prompt. For example, inputting a screenshot of a blue circular button, the model may output "blue circular save button for confirmation operation".

[0144] The above LVLM model can be obtained by fine-tuning a training data set. In the training data set, the input data of the model is: image data, a GUI screenshot containing a marked interactive element bounding box; a prompt text guiding the LVLM model to generate a description text. The label is: a standard answer annotated by humans, a complete description containing visual and functional information, such as "blue circular save button for confirmation operation".

[0145] As an example, the dataset contains an input image, a prompt text, and a label, where the input image is a screenshot of a browser interface with a search input box marked; the prompt text is "Please describe the UI element in the marked area"; and the label is "White rectangular input box containing placeholder text 'Search the web', used for entering query keywords".

[0146] Based on the training dataset, the LVLM model can be fine-tuned to adapt to the generation task of GUI element description. Specifically, the parameters of the LVLM model can be adjusted to minimize the difference (loss) between the LVLM model output and the manually annotated true description, in order to improve the accuracy of the generated description text. The loss function may, for example, be cross-entropy loss, which measures the degree of matching between the predicted text of the LVLM model and the label text, and optimizes the model through backpropagation.

[0147] In the embodiments of the present disclosure, through the end-to-end processing of the LVLM, the automatic generation from the original image to the semantic description is realized, and the generation efficiency and quality of the GUI data are significantly improved. By fine-tuning the LVLM model, the LVLM can adapt to the particularity of the GUI annotation data generation field. By using cross-platform and multi-scene training data, the model can learn the design language of different operating system platforms, such as distinguishing between the flat buttons of Windows and the skeuomorphic icons of macOS. This generalization ability makes the generated description text maintain high accuracy in different environments, such as correctly describing "green rectangular execution button" in a Linux terminal tool.

[0148] In addition, the text describing the target interactive element based on the LVLM output based on natural language provides directly usable multi-modal data for the subsequent training of GUI understanding models, such as intelligent assistants supporting natural language interaction can accurately parse the "click the blue circular save button" instruction, greatly improving the user interaction experience.

[0149] The following describes an application scenario of an embodiment of the present disclosure. In this application scenario, the GUI elements of a file management interface of a certain desktop software are annotated to generate GUI annotation data. Figure 2 A flowchart of another GUI data generation method according to an embodiment of the present disclosure is shown in FIG. 6. As shown in FIG. 6, the specific process is as follows: Figure 2

[0150] Step S21, obtaining a GUI image. Specifically, the file management interface screenshot can be obtained by an automated tool, which contains menu bar, folder list, operation button, etc., to obtain the original GUI image file.

[0151] ​Step S22, interactive element detection is performed on the GUI image. Specifically, non-text interactive elements can be identified by a visual detection model to obtain the positions of the non-text interactive elements:

[0152] a "New Folder" button (coordinates: [100, 200, 150, 250]);

[0153] a "Delete" icon (coordinates: [300, 400, 350, 450]);

[0154] a "Search" input box (coordinates: [500, 50, 700, 80]).

[0155] Text elements are identified by an OCR module, and the text elements specifically include:

[0156] a "Documents" folder label (coordinates: [120, 220, 180, 240]);

[0157] a "Quick Access" title (coordinates: [50, 30, 150, 50]).

[0158] Step S23, redundant elements are removed from the detected interactive elements. Specifically, the overlap ratio (IoU = 0.85) of the text "Documents" and the "New Folder" button is calculated, and it is determined that they are the same element, and the text detection box is removed; the text "Quick Access" does not overlap with any control, and it is determined that it is a non-interactive element, and is directly removed.

[0159] Step S24, spatial distribution sampling is performed on the interactive elements after removing the redundant elements to obtain target interactive elements. From the 10 detected interactive elements, 5 elements with discrete distribution are selected, the "New Folder" button is first randomly selected; then the "Search" input box that is farthest away is selected; the "Delete" icon that is the second farthest away is selected; the above process is iteratively executed, and the "Properties" button and the "Back" arrow are sequentially selected. The target interactive elements selected are marked in the GUI image.

[0160] Step S25, image analysis is performed on the target interactive elements to generate description text. The GUI image with the target interactive elements marked and the prompt text are input into the trained LVLM model, for example, the prompt text is "Please describe the UI elements in the marked area in the image". The text description output by the model is obtained: "0, red trash can icon, used for deleting selected files"; "1, white rectangular search box, containing placeholder text 'Search files'", and so on.

[0161] Step S26, the position information of the target interactive element is integrated with the generated description text to obtain GUI data. For example, "[300, 400, 350, 450], red garbage can icon, used for deleting selected files", "[500, 50, 700, 80], white rectangular search box, containing placeholder text'search file'", and the like.

[0162] So far, the GUI data is obtained, which can be used in subsequent GUI understanding models, such as voice interaction control, automatic testing, and the like.

[0163] In addition, the present disclosure also provides a graphical user interface GUI data generation apparatus, an electronic device, a computer readable storage medium, and a program, which can be used to implement any of the graphical user interface GUI data generation methods provided by the present disclosure. The corresponding technical solutions and descriptions are described in the method part and are not repeated here.

[0164] Figure 3 A block diagram of a graphical user interface GUI data generation apparatus according to an embodiment of the present disclosure is shown as follows. Figure 3 As shown in the figure, the apparatus 30 includes:

[0165] A detection module 31 is configured to perform interactive element detection on a GUI image to obtain position information of a target interactive element.

[0166] A description text generation module 32 is configured to perform image analysis on the target interactive element in the GUI image based on the position information of the target interactive element to generate a description text. The description text is used to describe visual information and functional information of the target interactive element.

[0167] An integration module 33 is configured to integrate the position information of the target interactive element with the description text to obtain GUI data.

[0168] In a possible implementation, the detection module is configured to:

[0169] Identify non-text interactive elements and text elements in the GUI image.

[0170] According to the spatial relationship and the degree of overlap between the non-text interactive elements and the text elements, redundant elements are removed to obtain the position information of the target interactive element.

[0171] In a possible implementation, the detection module is configured to:

[0172] In a case where the detection box of the text element is contained in the detection box of the non-text interactive element, the text element is determined as a redundant element and is removed.

[0173] In a case where the bounding box of the text element is completely located outside the bounding box of the non-text interactive element, the text element is determined as a non-interactive element and is removed.

[0174] In a possible implementation, the detection module is configured to:

[0175] calculate an overlap ratio of the bounding boxes of the text element and the non-text interactive element, the overlap ratio being a ratio of an intersection area to a union area of the two bounding boxes;

[0176] In a case where the overlap ratio is lower than a preset threshold, the text element is determined as a non-interactive element and is removed.

[0177] In a possible implementation, the detection module is configured to:

[0178] perform spatial distribution sampling on the detected interactive elements, and select interactive elements with a distributed discrete as target interactive elements.

[0179] In a possible implementation, the detection module is configured to:

[0180] randomly select an initial interactive element from the detected interactive elements;

[0181] calculate distances between remaining unselected elements and a selected element selected last time, and randomly select an unselected element from a first N unselected elements with the largest distances; N is a positive integer.

[0182] repeat the process of calculating and selecting an unselected element until a preset sampling number of selected elements is reached.

[0183] In a possible implementation, the description text generation module is configured to

[0184] perform visual analysis and semantic analysis on the target interactive element in the GUI image based on position information of the target interactive element, to obtain visual information and function information of the target interactive element.

[0185] generate a description text based on the visual information and the function information.

[0186] In a possible implementation, the description text generation module is configured to

[0187] input the GUI image containing position information of the target interactive element and the prompt text into a large-scale visual language model (LVLM), to obtain a description text of the target interactive element; the description text is a text describing the target interactive element based on natural language.

[0188] In a possible implementation, the GUI image is a cross-platform desktop application screenshot.

[0189] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0190] The embodiments of the present disclosure also provide an electronic device, including a memory, a processor and a computer program stored in the memory, the processor executes the computer program to implement the steps of the above method.

[0191] The embodiments of the present disclosure also provide a non-volatile computer readable storage medium having a computer program stored thereon, the computer program is executed by a processor to implement the steps of the above method.

[0192] The embodiments of the present disclosure also provide a computer program product, including a computer program or a non-volatile computer readable storage medium carrying a computer program, the computer program is executed by a processor to implement the steps of the above method.

[0193] Figure 4 is a block diagram of an apparatus 1900 for graphical user interface (GUI) data generation according to an exemplary embodiment. For example, the apparatus 1900 can be provided as a server or a terminal device. Referring to Figure 4 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0194] The apparatus 1900 can also include a power supply component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input output interface 1958 (I / O interface). The apparatus 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0195] In an example embodiment, there is also provided a non-transitory computer- readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component(s) 1922 of the apparatus 1900 to implement the above-described methods.

[0196] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0197] The computer programs (or computer readable program instructions) described herein can be downloaded from a computer readable storage medium to the respective computing / processing devices, or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network, and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0198] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0199] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0200] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0201] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0202] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0203] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A Graphical User Interface (GUI) data generation method, characterized by, The method comprises the following steps: interactive element detection is performed on a GUI image to obtain position information of a target interactive element; image analysis is performed on the target interactive element in the GUI image based on the position information of the target interactive element to generate a description text; the description text is used to describe visual information and functional information of the target interactive element; the position information of the target interactive element is integrated with the description text to obtain GUI data, and the GUI data is used to train a GUI understanding model; the interactive element detection on the GUI image to obtain the position information of the target interactive element comprises spatial distribution sampling on the detected interactive elements, and the interactive elements with discrete distribution are selected as the target interactive elements.

2. The method of claim 1, wherein, The interactive element detection on the GUI image to obtain the position information of the target interactive element comprises the following steps: non-text interactive elements and text elements in the GUI image are identified; redundant elements are removed according to spatial relationships and overlapping degrees between the non-text interactive elements and the text elements to obtain the position information of the target interactive elements.

3. The method of claim 2, wherein, The removal of the redundant elements according to the spatial relationships and the overlapping degrees between the non-text interactive elements and the text elements to obtain the position information of the target interactive elements comprises the following steps: in a case where a detection box of the text element is contained in a detection box of the non-text interactive element, the text element is determined as a redundant element and is removed; in a case where the detection box of the text element is completely located outside the detection box of the non-text interactive element, the text element is determined as a non-interactive element and is removed.

4. The method of claim 2, wherein, The removal of the redundant elements according to the spatial relationships and the overlapping degrees between the non-text interactive elements and the text elements to obtain the position information of the target interactive elements comprises the following steps: an overlapping ratio of the detection boxes of the text element and the non-text interactive element is calculated, the overlapping ratio being a ratio of an intersection area to a union area of the two detection boxes; in a case where the overlapping ratio is lower than a preset threshold, the text element is determined as a non-interactive element and is removed.

5. The method of claim 1, wherein, The spatial distribution sampling on the detected interactive elements to select the interactive elements with discrete distribution as the target interactive elements comprises the following steps: an initial interactive element is randomly selected from the detected interactive elements; a distance between a remaining unselected element and a selected element selected last time is calculated, and an unselected element is randomly selected from the first N unselected elements with the largest distance; N is a positive integer; the process of the calculation and the selection of the unselected element is repeatedly iterated until the selected elements reach a preset sampling number.

6. The method of claim 1, wherein, The image analysis on the target interactive element in the GUI image based on the position information of the target interactive element to generate the description text comprises the following steps: visual analysis and semantic analysis are performed on the target interactive element in the GUI image based on the position information of the target interactive element to obtain visual information and functional information of the target interactive element; the description text is generated based on the visual information and the functional information.

7. The method of claim 1, wherein, The image analysis on the target interactive element in the GUI image based on the position information of the target interactive element to generate the description text comprises the following steps: The GUI image containing the position information of the target interactive element and prompt text are input into a large-scale visual language model (LVLM) to obtain a description text of the target interactive element; the description text is a text describing the target interactive element based on natural language.

8. The method of claim 1, wherein, The GUI image is a cross-platform desktop application screenshot.

9. A Graphical User Interface (GUI) data generating apparatus characterized by comprising: The method comprises the steps of: a detection module configured to detect interactive elements in the GUI image to obtain position information of a target interactive element; a description text generation module configured to perform image analysis on the target interactive element in the GUI image based on the position information of the target interactive element to generate a description text; the description text is used to describe visual information and functional information of the target interactive element; an integration module configured to integrate the position information of the target interactive element with the description text to obtain GUI data, wherein the GUI data is used to train a GUI understanding model. The detection module is configured to: perform spatial distribution sampling on the detected interactive elements, and select interactive elements with discrete distribution as target interactive elements.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 9. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 8.

11. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 8.

Images