Data acquisition method and system

Abstract

The invention provides a data acquisition method and system, and the method comprises the steps: initializing a dynamic graph (including an initial node corresponding to a home page of an application program, and the initial node comprises a screenshot and a control list of the home page), repeating the following steps until a target dynamic graph meeting a preset condition is obtained, and determining data for training a GUI Agent according to the target dynamic graph. The preset condition comprises that each control of the target dynamic graph is an accessed control, and the repeated steps comprise: determining a current non-accessed control from a control list of a current node, determining and executing a current operation corresponding to the current non-accessed control to obtain a jump page, and updating the current dynamic graph according to the jump page, the updated dynamic graph comprises a first node corresponding to the jump page and a current edge between the current node and the first node, and updating a current unaccessed control based on the unaccessed control in a control list of the first node. The effectiveness and reliability of data acquisition can be improved.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence (AI) technology, and in particular to a data acquisition method and system. Background Technology

[0002] With the widespread application of large-model-driven graphical user interface (GUI) agents in scenarios such as mobile phone automation and digital life assistants, high-quality, large-scale training data has become a key bottleneck restricting their performance.

[0003] In related technologies, training data for training GUI agents is mainly collected through the following three methods: Method 1): Collecting training data through manual annotation. For example, annotators obtain and annotate operation trajectories according to predefined task objectives to obtain training data. However, this method requires annotators to have a considerable degree of domain knowledge, and the annotation cost is high, time-consuming, and repetitive. In addition, this method has limited coverage and is difficult to extend to long-tail applications.

[0004] Method 2): Obtaining training data through task-driven synthesis. For example, relying on predefined task objectives, a large language model performs inference and execution based on the task objectives and the user interface (UI) to obtain training data. However, the data quality of this method depends entirely on the planning capabilities of the large language model itself, and it suffers from insufficient task diversity.

[0005] Method 3): Obtaining training data through interaction-driven methods. For example, randomly interacting with controls in the UI automatically generates a large number of operation trajectories to obtain training data. However, the randomization process in this method may be too divergent and difficult to converge to a valid intent. In addition, it may also lead to serious redundancy in operation trajectories and incomplete interaction exploration.

[0006] It should be noted that the above-mentioned related technologies are only information known to the inventor personally, and do not mean that the above information had entered the public domain before the application date of this specification, nor do they mean that it can be considered prior art in this specification. Summary of the Invention

[0007] This specification provides a data acquisition method and system to avoid at least one of the above-mentioned technical problems.

[0008] Firstly, this specification provides a data acquisition method, including: An initialization animation of the application is provided, wherein the initialization animation includes a starting node corresponding to the application's homepage, and the starting node includes a screenshot of the homepage and a list of controls; Repeat the following steps until a target animated image that meets the preset conditions is obtained, wherein the preset conditions include that all controls in the control list of each node in the target animated image are accessed controls: Identify currently unaccessed controls from the control list of the current node; Determine and execute the current operation corresponding to the currently unaccessed control to obtain the redirected page; The current dynamic graph is updated according to the redirected page, wherein the updated dynamic graph includes a first node corresponding to the redirected page and a current edge between the current node and the first node. The first node includes a screenshot of the redirected page and a list of controls, and the current edge represents the current operation. Update the currently unaccessed controls based on the unaccessed controls in the control list of the first node; Training data for training the GUI Agent is determined based on the target dynamic graph that meets the preset conditions.

[0009] Secondly, this specification provides a data acquisition system, including: At least one storage medium storing at least one instruction set for data acquisition; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, it reads the at least one instruction set and executes the method as described in the first aspect according to the instructions of the at least one instruction set.

[0010] Thirdly, this specification provides a computer-readable non-transitory storage medium, wherein the computer-readable non-transitory storage medium stores at least one instruction set, which is executed by at least one processor to implement the method as described in the first aspect.

[0011] As can be seen from the above technical solutions, the data acquisition method and system provided in this specification start from the application's homepage and dynamically model the transition relationships between UI states during the data acquisition process, constructing and gradually expanding a dynamic graph reflecting the UI interaction structure of the application. Furthermore, during the exploration process of dynamic modeling, operations to be executed are dynamically selected based on unvisited controls on the current page to guide the exploration towards uncovered areas until all controls in the dynamic graph are marked as visited. Based on this dynamic graph, a serialized trajectory containing UI states and corresponding operations is further extracted as training data for training the GUI Agent. This achieves fully automated training data acquisition; it also ensures that the training data closely resembles actual user interaction scenarios, thereby improving the generalization ability and robustness of the trained GUI Agent in real-world environments; it significantly expands the number of training samples, ensuring data sufficiency; it effectively improves the coverage and validity of the UI state transition relationship space; and it makes the process of obtaining training data more efficient and has a wider coverage.

[0012] The data acquisition methods and other functions of the system provided in this specification are partially listed in the following description. The inventive aspects of the data acquisition methods and systems provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram illustrating an application scenario of the data acquisition method provided in the embodiments of this specification; Figure 2 This is a schematic diagram of the data acquisition system provided in the embodiments of this specification; Figure 3 A flowchart illustrating the data acquisition method provided in the embodiments of this specification; Figure 4 This is a schematic diagram of page navigation provided in the embodiments of this specification; Figure 5 This is a schematic diagram illustrating the construction of the dynamic diagram provided in the embodiments of this specification; Figure 6 This is a schematic diagram illustrating the principle of determining a currently unaccessed control as provided in the embodiments of this specification.

[0015] Figure 7This is a schematic diagram illustrating how two not-quite-same pages are identified as the same page, provided for an embodiment of this specification. Detailed Implementation

[0016] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0017] It should be understood that the terms “comprising” and “having”, and any variations thereof, in the embodiments of this specification are intended to cover but not exclude inclusion. For example, a product or device that includes a series of components is not necessarily limited to those components that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0018] The term "and / or" in the embodiments of this specification describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0019] In the embodiments of this specification, the term "multiple" refers to two or more, and other quantifiers are similar.

[0020] The terms “first,” “second,” “third,” etc., used in this specification are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, in situations where implementation can proceed in an order other than those given in the embodiments illustrated or described in this specification.

[0021] As used in this specification, the term "unit / module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0022] To facilitate the reader's understanding of this manual, at least some of the terms used in this manual are explained below: Large Language Models (LLMs), also known simply as large models, are natural language processing models based on deep learning techniques. Their parameter count typically ranges from billions to hundreds of billions or even higher, possessing powerful language understanding and generation capabilities. LLMs can employ the Transformer architecture or its variants (such as GPT and BERT), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. By pre-training on large-scale corpora, LLMs learn the statistical features and semantic relationships of language, enabling them to achieve outstanding generalization capabilities. The core capabilities of LLMs include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Their usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, users design prompts to guide the LLM in generating specific outputs. Prompts can be textual descriptions of the task or instructions used to stimulate the LLM's semantic understanding and generation capabilities. In fine-tuning mode, large language models are further trained on small-scale datasets within a specific domain to optimize their performance on specific tasks. The powerful generalization capabilities and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0023] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0024] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0025] A GUI (User Interface) is a visual interface through which users interact with electronic devices (such as smartphones, tablets, and computers). It typically consists of operable controls such as windows, icons, buttons, text boxes, and menus, allowing users to complete tasks through clicking, swiping, and inputting. In this specification, GUI primarily refers to the screen interface state of mobile applications (Apps).

[0026] A GUI Agent is an intelligent agent for graphical user interfaces. It is an AI system that can understand screen content and perform operations such as clicking and swiping to complete user tasks.

[0027] A widget (or UI element) is an interactive element on a UI, such as a button, input field, or tab bar.

[0028] An action trajectory is an ordered sequence of actions (such as clicking, swiping, and inputting) performed by a user or automated agent starting from an initial UI state, along with the corresponding UI state transition sequence.

[0029] To avoid at least one of the technical problems mentioned in the background section, this specification proposes a technical concept developed through inventive effort: starting from the application's initial page, the state transition relationships of the application's UI pages are dynamically modeled to obtain a dynamic graph including nodes and edges. Nodes represent UI states of the UI page, such as screenshots and control lists, and edges between two nodes represent actions that transition from one UI state (corresponding node) to another UI state (corresponding node) through a control operation. Furthermore, during the dynamic modeling process, the operation to be performed is dynamically selected based on the number of unaccessed controls in the UI until all controls in the final dynamic graph are accessed, thereby determining the sample data used to train the GUI Agent based on the final dynamic graph.

[0030] For example, for the application's homepage, an animated graph is initialized to construct a node corresponding to the homepage (for ease of distinction, this can be called the starting node). Based on the above analysis, a node can represent the UI state. Therefore, the starting node can represent the UI state of the homepage; for example, the starting node can include a screenshot of the homepage and a list of controls (including all controls on the homepage).

[0031] It's important to understand that UI state can also include other information, such as UI page features (vectors). These page features can represent different characteristics of the UI from different dimensions. For example, page features can include image features representing the UI's image content from an image dimension, text features representing the UI's text content from a text dimension, and layout features representing the UI's layout content from a layout dimension.

[0032] The controls in the control list of the starting node may be accessed controls or unaccessed controls. The number of unaccessed controls may be one or more, and during initialization, the number of unaccessed controls is usually multiple.

[0033] For multiple unaccessed controls, the steps of dynamically building the dynamic graph can be repeated until all controls in the final dynamic graph are accessed controls.

[0034] For example, during a certain execution step, an unaccessed control is first identified from the control list of the current node, and the corresponding operation is performed to obtain the new UI after the operation. Then, the node of the new UI (including the screenshot of the new UI and the control list) is constructed in the current dynamic graph, and the edge between the current node and the new UI node (the edge represents the operation performed) is constructed. Finally, the unaccessed control in the control list of the new UI node is taken as the current unaccessed control.

[0035] This process continues until a dynamic graph with no unaccessed controls is obtained.

[0036] The technical solution provided in this specification is based on the aforementioned technical concept. As described above, the technical solution provided in this specification starts from the application's homepage and dynamically models the transition relationships between UI states during data collection, constructing and gradually expanding a dynamic graph reflecting the UI interaction structure of the application. Furthermore, during the exploration process, operations to be performed are dynamically selected based on unvisited controls on the current page to guide the exploration towards uncovered areas until all controls in the dynamic graph are marked as visited. Based on this dynamic graph, a serialized trajectory containing UI states and corresponding operations is further extracted as training data for training the GUI Agent. This solution has the following technical advantages: (1) It realizes the full automation of training data collection, avoiding the high cost and low efficiency of the traditional method 1 which relies on manually writing scripts or rules for annotation; (2) Since the dynamic graph is constructed in real time based on the real UI state and its operation feedback, the generated training data is highly close to the actual user interaction scenario, which helps to improve the generalization ability and robustness of the GUI Agent in the real environment; (3) Compared with the problem of insufficient data scale caused by the reliance on limited manual demonstration in Method 2, this solution can significantly expand the number of training samples and ensure the sufficiency of data; (4) Compared with the problems of low exploration efficiency, path redundancy and uneven coverage caused by the random click strategy in Method 3, this solution effectively improves the coverage of the state space and the effectiveness of the data through the heuristic exploration mechanism guided by unvisited controls.

[0037] To facilitate readers' understanding of this manual, the application scenarios of this manual are introduced below.

[0038] The technical solutions provided in this specification are applicable to scenarios that require the collection of training data for training GUI Agents. For example, a GUI Agent can be applied to e-commerce application scenarios, and the application can be an e-commerce application within an e-commerce application scenario; another example is that a GUI Agent can be applied to social application scenarios, and the application can be a social application within a social application scenario; yet another example is that a GUI Agent can be applied to content and information application scenarios, and the application can be a content and information application within a content and information application scenario; yet another example is that a GUI Agent can be applied to tool application scenarios, and the application can be a tool application within a tool application scenario; and so on, and will not be listed here.

[0039] For example, let's take the technical solution provided in this specification as an example when applied to an e-commerce application scenario: Figure 1 This is a schematic diagram illustrating an application scenario of the data acquisition method described in this specification. The data acquisition method described in this specification can be applied to, for example... Figure 1 Scenario 100 is shown. (e.g.) Figure 1 As shown, scenario 100 may include target user 101 and client 102. Client 102 deploys an e-commerce application (specifically a food delivery application) and a corresponding GUIAgent.

[0040] In scenario 100, the scenario involves a target user 101 and a client 102 configured with a corresponding e-commerce application (taking a food ordering application as an example) and a corresponding GUI Agent. The target user 101 can make a food order request to the GUI Agent via voice commands, such as: "Order a braised chicken rice dish, delivered to the office." Based on this command, the GUI Agent can simulate the human user's workflow within the food ordering application to complete the order processing.

[0041] like Figure 1 As shown, the specific steps of the GUI Agent to complete order processing are as follows: First, the GUI Agent clicks the search box on the homepage of the food ordering app. Next, the GUI Agent enters "braised chicken rice" in the activated search box and executes the search command. Then, the GUI Agent locates the "XX braised chicken" option provided by a specific merchant in the displayed product list and clicks to enter the product details page. On this page, the GUI Agent adds the required quantity of the product by clicking the "+" button and continues to the checkout page. On the order confirmation page, the GUI Agent selects "Company" as the delivery address. Finally, the GUI Agent clicks the "Pay Now" button to complete the entire order process.

[0042] In comparison, this automated operation method not only improves the convenience of user experience, but also verifies the interactive capabilities and accuracy of GUIAgent in real-world application scenarios.

[0043] It should be noted that the above examples are only used to illustrate the application scenarios to which the technical solutions in this specification can be applied, and should not be construed as limiting the application scenarios.

[0044] Figure 2 A hardware structure diagram of a data acquisition system 200 according to an embodiment of this specification is shown. The data acquisition system 200 can perform the data acquisition methods described in this specification. The data acquisition methods are described in other parts of this specification.

[0045] like Figure 2 As shown, the data acquisition system 200 may include at least one storage medium 203 and at least one processor 202. In some embodiments, the data acquisition system 200 may also include a communication port 204 and an internal communication bus 201. The data acquisition system 200 may also include I / O components 205.

[0046] The internal communication bus 201 can connect to different system components. For example, the internal communication bus 201 can connect to storage medium 203, processor 202, communication port 204, and I / O component 205.

[0047] I / O component 205 supports input / output between data acquisition system 200 and other components.

[0048] Communication port 204 is used for data communication between the data acquisition system 200 and external sources. For example, communication port 204 can be used for data communication between the data acquisition system 200 and a network. Communication port 204 can be a wired communication port or a wireless communication port.

[0049] Storage medium 203 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 2031, a read-only storage medium (ROM) 2032, or a random access storage medium (RAM) 2033. Storage medium 203 also includes at least one instruction set stored in the data storage device. The instruction set includes computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the data acquisition methods provided in this specification.

[0050] At least one processor 202 may be communicatively connected to at least one storage medium 203. The at least one processor 202 is used to execute the at least one instruction set described above. When the data acquisition system 200 is running, the at least one processor 202 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the data acquisition method provided in this specification. The processor 202 may execute all steps included in the data acquisition method. The processor 202 may be in the form of one or more processors. In some embodiments, the processor 202 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof.

[0051] For illustrative purposes only, only one processor 202 is shown in the accompanying drawings of the data acquisition system 200. However, it should be noted that the data acquisition system 200 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor or by multiple processors in combination. For example, if the processor 202 of the data acquisition system 200 is described in this specification as executing steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 202 (e.g., the first processor executes step A, the second processor executes step B, or the first and second processors jointly execute steps A and B).

[0052] Please see Figure 3 , Figure 3 This is a flowchart illustrating the data acquisition method provided in the embodiments of this specification. Wherein, Figure 3 The data acquisition method shown can be implemented by a data acquisition system. For a description of the data acquisition system, please refer to the example above; it will not be repeated here.

[0053] like Figure 3 As shown, the method includes the following steps S301 to S303: S301: Initialize the application's dynamic graph, wherein the initialized dynamic graph includes a starting node corresponding to the application's home page, and the starting node includes a screenshot of the home page and a list of controls.

[0054] A dynamic graph is a graph that dynamically models the UI state transitions of an application, starting from the application's initial page.

[0055] Accordingly, the initial dynamic graph is a model of the application's homepage. This graph includes nodes representing the homepage state (referred to as the starting node for easy distinction). The homepage state can include screenshots of the homepage and a list of controls within the homepage.

[0056] The homepage refers to the interface first presented to the user after the application starts; the control list refers to the collection of all interactive elements on the application interface, such as buttons and input boxes. Therefore, the control list on the homepage refers to the collection of all interactive elements in the interface first presented to the user after the application starts.

[0057] For example, let's continue with the above-mentioned food delivery application: For the homepage displayed by the food delivery application, the data collection system can collect its image (such as by recording) to obtain a screenshot of the homepage.

[0058] The data acquisition system can also identify controls on the homepage to obtain a list of controls on the homepage.

[0059] In addition, for each control in the homepage control list, the data collection system can maintain its access status using tags. For example, if a control's tag is 0, it means that the control has not been accessed; if a control's tag is 1, it means that the control has been accessed. An unaccessed control is one that has not been clicked or interacted with.

[0060] S302: Repeat steps 21 to 24 until a target dynamic graph that meets the preset conditions is obtained, wherein the preset conditions include that all controls in the control list of each node in the target dynamic graph are accessed controls.

[0061] Understandably, initially, the controls in the homepage's control list are all unaccessed controls. However, as data collection continues and dynamic graphs are continuously modeled, the unaccessed controls in the homepage's control list will gradually become accessed controls.

[0062] Furthermore, as data collection continues, the homepage may redirect to other pages, and other pages may also redirect. Therefore, the target dynamic graph may include multiple nodes, and different nodes may have different transition relationships.

[0063] In this embodiment, it is equivalent to using unaccessed controls in the control list as a guide to search for the transition relationships of the application's UI state in order to dynamically model and obtain a target dynamic graph that meets preset conditions.

[0064] Step 21: Identify currently unaccessed controls from the current node's control list.

[0065] Step 22: Identify and execute the current operation corresponding to the currently unaccessed control to get the redirected page.

[0066] Step 23: Update the current dynamic graph according to the redirected page. The updated dynamic graph includes the first node corresponding to the redirected page and the current edge between the current node and the first node. The first node includes a screenshot of the redirected page and a list of controls. The current edge represents the current operation.

[0067] Step 24: Update the currently unaccessed controls based on the unaccessed controls in the control list of the first node.

[0068] For example, S301 can be understood as the initialization phase, and steps 21 to 24 in S302 can be understood as the exploration loop phase to generate the target dynamic graph.

[0069] For example, the exploration loop phase can be understood as continuously selecting one unvisited control to perform an operation on, recording the resulting new page (the page that jumps to), and the edges connecting the previous and next pages (representing the operation performed). This process is repeated until all controls in the target dynamic graph have been visited.

[0070] In some embodiments, if the repeated steps are being performed for the first time, the currently unaccessed control is the unaccessed control determined from the control list on the homepage.

[0071] For example, after the initialized dynamic graph is constructed, when step 21 above is executed for the first time, the data acquisition system determines the unaccessed controls from the control list on the homepage as the currently unaccessed controls.

[0072] Specifically, let's continue with food delivery apps as an example: like Figure 4 and Figure 5 As shown, if the current node is the starting node corresponding to the homepage of the food delivery application, and the control list of the current node includes control 1 and control 2, then the current node includes a screenshot of the homepage and a control list including control 1 and control 2. Assume that control 1 and control 2 are both unaccessed controls (i.e., controls that have not been accessed before), and control 1 is a "search box" and control 2 is a "category browsing" button.

[0073] Therefore, the data acquisition system can determine the currently unaccessed control from control 1 and control 2. For example, the data acquisition system can determine the currently unaccessed control from control 1 and control 2 based on preset priority rules, preset control type weights, preset order, etc. For example, ... Figure 4 As shown, assume that the data acquisition system treats control 1 as a currently unaccessed control.

[0074] Accordingly, the current operation can be: click control 1, and enter the keyword "braised chicken rice" in the pop-up search input bar and press Enter (or click the "search" button).

[0075] Continue reading Figure 4 The page you are redirected to after performing the current operation will be the search results page. The search results page may display a list of businesses that offer "braised chicken rice" (such as "XX Braised Chicken", "Lao Zhang Braised Chicken", etc.).

[0076] The data acquisition system can update the current animated graph. For example, suppose the current animated graph is the initial animated graph: On the one hand, such as Figure 5As shown, the data acquisition system can construct node 1 corresponding to the search results page in the initialized dynamic graph. Node 1 includes a screenshot of the search results page and a list of controls including controls in the search results page (such as control 11 "XX Braised Chicken", control 12 "Old Zhang Braised Chicken", etc.).

[0077] On the other hand, such as Figure 5 As shown, the data system can also construct an edge between the starting node and node 1. For example... Figure 5 As shown, the edges can be directed edges to represent the direction of the jump. For example, this edge represents clicking control 1 on the homepage, entering the keyword "braised chicken rice" in the pop-up search input bar, and pressing Enter (or clicking the "Search" button) to jump to the search results page.

[0078] Furthermore, the data acquisition system can update the currently unaccessed controls based on the unaccessed controls in the control list of node 1.

[0079] For example, suppose the unaccessed controls in the control list of node 1 include control 11—“XX Braised Chicken” and control 12 “Old Zhang Braised Chicken”. The data acquisition system can then update the currently unaccessed control—control 1—based on control 11 and control 12. If control 11 is set as the currently unaccessed control, the above operation can be performed again.

[0080] Assuming that control 11 is currently not accessed, the current operation on control 11 can be clicking, which will redirect to the product details page of XX Braised Chicken.

[0081] Similarly, the data acquisition system can update the current dynamic graph. At this point, the current dynamic graph is a graph that includes the starting node, node 1, and the edge between the starting node and node 1.

[0082] Then as Figure 5 As shown, on the one hand, the data acquisition system can construct node 11 corresponding to the XX Braised Chicken product details page in the current dynamic graph. Node 11 includes a screenshot of the XX Braised Chicken product details page and a list of controls including those found on the XX Braised Chicken product details page. Assuming there are no controls on the XX Braised Chicken product details page, then... Figure 5 As shown, the control list for node 11 can be empty.

[0083] On the other hand, such as Figure 5 As shown, the data acquisition system can also construct an edge between node 1 and node 11, which represents the click control 11.

[0084] In some embodiments, if there are no unaccessed controls in the control list of the first node (e.g., all controls in the control list of the first node are accessed controls, or there are no controls in the control list of the first node), then the currently unaccessed controls are updated according to the unaccessed controls in the control list of the parent node of the first node.

[0085] Continuing with the examples above and Figure 5 If the control list of node 11 is empty, that is, node 11 has no child nodes, then after constructing node 11 and the edge between node 1 and node 11 (i.e., the dynamic graph has been updated), the data collection can return to the parent node of node 11—node 1—to determine whether all the controls in the control list of node 1 are accessed controls.

[0086] If so (i.e., all controls in the control list of node 1 are accessed), then return to the parent node of node 1—the starting node, and so on, until all controls in the control list of all nodes are accessed.

[0087] If not (i.e., at least one control in the control list of node 1 is still an unaccessed control), such as control 12 being an unaccessed control, the data acquisition system can update the currently unaccessed control—control 11—based on control 12. If control 12 is set as the currently unaccessed control, then the above operation is performed again.

[0088] Assuming that control 12 is currently not accessed, the current operation on control 12 can be clicking, which will redirect to the product details page of Lao Zhang Braised Chicken.

[0089] Similarly, the data acquisition system can update the current dynamic graph. At this time, the current dynamic graph is a graph that includes the starting node, node 1, node 11, the edge between the starting node and node 1, and the edge between node 1 and node 11.

[0090] Then as Figure 5 As shown, on the one hand, the data acquisition system can construct node 12 corresponding to the Lao Zhang Braised Chicken product details page in the current dynamic graph. Node 12 includes a screenshot of the Lao Zhang Braised Chicken product details page and a list of controls including those found on the Lao Zhang Braised Chicken product details page. Assuming there are no controls on the Lao Zhang Braised Chicken product details page, then... Figure 5 As shown, the control list for node 12 can be empty.

[0091] Correspondingly, if the control list of node 12 is empty, the control list of control 11 is empty, and all controls in the control list of node 1 are accessed controls, then the data acquisition system can update the currently unaccessed controls based on the unaccessed controls in the control list of the parent node - the starting node of node 1.

[0092] For example, the data acquisition system can treat control 2 as an unaccessed control and perform the above operations (to avoid tedious details, they will not be listed here) until the target dynamic image is obtained.

[0093] Furthermore, the parent node in this specification can be a first-level parent node. For example, after all nodes in the control list of node 11 are accessed controls, the currently unaccessed control is determined from the unaccessed controls in the control list of its first-level parent node (node ​​1). The parent node in this specification can be a multi-level parent node. For example, taking a second-level parent node as an example, after all nodes in the control list of node 11 are accessed controls, the currently unaccessed control is determined from the unaccessed controls in the control list of its second-level parent node (starting node).

[0094] In this specification, we typically start from the first-level parent node and work backwards sequentially to avoid data omissions and improve comprehensiveness and effectiveness.

[0095] In this embodiment, updating currently unaccessed controls by using the association between the current node and its parent node can improve the accuracy, reliability, and efficiency of updating currently unaccessed controls.

[0096] Based on the above analysis, it can be seen that this embodiment does not limit the method for determining currently unaccessed controls. This specification proposes a method to proactively assess the potential information space size of each control and avoid excessive time consumption.

[0097] For example, step 21 above may include: determining the set of reachable pages after executing the unaccessed controls in the control list of the current node; for each reachable page in the set of reachable pages, determining the number of unaccessed controls in that reachable page; and determining the unaccessed control in the control list of the current node with the largest number of unaccessed controls as the currently unaccessed control.

[0098] The reachable page set refers to the collection of new UI elements that are actually redirected to or presented after performing the corresponding operation on each unvisited control under the current node.

[0099] For example, continuing to combine the above examples and Figure 6 Assuming that under the current node, the unvisited controls include control 1 and control 2, the data acquisition system can determine that clicking control 1 leads to page 1 and clicking control 2 leads to page 2, thus obtaining a set that includes both page 1 and page 2.

[0100] Based on this, the data acquisition system can determine the number of unaccessed controls in reachable page 1 as XX, and the number of unaccessed controls in reachable page 2 as YY. The data acquisition system can then compare the number of unaccessed controls XX with the number of unaccessed controls YY to determine their relative size.

[0101] If the number of unaccessed controls XX is greater than the number of unaccessed controls YY, the data acquisition system can identify control 1 as the currently unaccessed control; if the number of unaccessed controls XX is less than the number of unaccessed controls YY, the data acquisition system can identify control 2 as the currently unaccessed control; if the number of unaccessed controls XX is equal to the number of unaccessed controls YY, the data acquisition system can randomly identify control 1 or control 2 as the currently unaccessed control, or it can use other methods to identify control 1 or control 2 as the currently unaccessed control, which is not limited in this embodiment.

[0102] In this embodiment, the data acquisition system determines currently unaccessed controls based on the number of unaccessed controls on the reachable page. This allows for concentrated resource exploration of relatively high-access areas and guides the exploration towards deeper and more branching pages, reducing omissions. Consequently, it enables proactive assessment of the potential information space for each control and avoids excessively time-consuming processes.

[0103] In some embodiments, each node in the current dynamic graph also includes page features of the page corresponding to that node.

[0104] For example, the page features of a page can be a structured or vectorized representation of the semantic visual state of the page's images. The page features of a page can be obtained by fusing the page's image features and text features.

[0105] For example, the current dynamic graph may include multiple nodes, and each node can represent the UI state of the corresponding page. Specifically, the current dynamic graph includes a starting node, which may include a screenshot of the homepage or a list of controls including all controls on the homepage. Additionally, the starting node may include page features of the homepage, which may include image features representing the image content of the homepage from an image dimension and / or text features representing the text content of the homepage from a text dimension, such as a concatenated feature including these image and text features.

[0106] Image features refer to numerical vectors obtained by performing computer vision processing (such as using CNNs to extract high-level semantic features) on screenshots of corresponding pages. These vectors can reflect visual information such as the layout, color distribution, and icon types of the corresponding page. For example, the image features of the homepage can reflect visual information such as the homepage layout, color distribution, and icon types.

[0107] Text features refer to the semantic representation formed by natural language processing (such as bag-of-words, TF-IDF, or embedding vectors) of all readable text extracted from a corresponding page (including control text, label text, button text, etc.), which can reflect the functional intent of the corresponding page. For example, the text features of the homepage can reflect the functional intent of the homepage.

[0108] Step 23 above may include the following sub-steps 1 to 4: Sub-step 1: Obtain the UI state of the redirected page, which includes a screenshot of the redirected page and a list of controls.

[0109] Continuing with the examples above and Figure 4 If the redirected page is the product details page for XX Braised Chicken, the UI of that page includes a screenshot of the XX Braised Chicken product details page, as well as a list of controls that include all controls on the XX Braised Chicken product details page.

[0110] Sub-step 2: Extract jump page features based on the UI state of the jump page. Jump page features include image features and text features of the jump page.

[0111] This embodiment does not limit the method for extracting features from redirected pages. For example, the data acquisition system can extract corresponding image features (text features) from a screenshot of the XX Braised Chicken product details page, and extract text features from the control list of the XX Braised Chicken product details page.

[0112] Specifically, the data acquisition system can use pre-trained visual models (such as ResNet, MobileNet, etc.) to extract relevant image features from screenshots of the XX Braised Chicken product details page. The system can first aggregate all text content from the control list of the XX Braised Chicken product details page and generate text features using lightweight NLP methods (such as keyword hashing, Sentence-BERT, etc.). Based on this, the system can concatenate or weightedly fuse the two into a feature vector, thereby obtaining the page features of the XX Braised Chicken product details page.

[0113] Alternatively, the data acquisition system can input the corresponding screenshots into the image encoder to obtain the corresponding image features; the data acquisition system can input the text from the corresponding control list (which may also include the text in the corresponding screenshots) into the text encoder to obtain the corresponding text features.

[0114] Sub-step 3: Match the characteristics of the redirected page with the page characteristics of each node in the current dynamic graph.

[0115] Continuing with the examples above, Figure 4 , Figure 5 The current dynamic graph includes a starting node and node 1. The data acquisition system can traverse the page features of the starting node and node 1 to calculate the similarity (specifically, cosine similarity) between the page features of the XX Braised Chicken product details page and the page of each existing node, so as to perform page feature matching.

[0116] Sub-step 4: If no existing node is matched, create the first node in the current dynamic graph based on the UI state of the page being navigated to, and create the current edge.

[0117] For example, there are usually two matching results: one is that no existing result is matched.

[0118] Continuing with the above example, if the matching result indicates that the page features of the XX Braised Chicken product details page are less similar to the page features of the starting node and the page features of node 1 (e.g., less than the preset similarity threshold), then the page features of the XX Braised Chicken product details page are different from the homepage of the starting node and also different from the search results page of node 1.

[0119] Therefore, the data acquisition system can determine that among the existing nodes in the current dynamic graph, no node corresponding to the XX Braised Chicken product details page has been matched. In other words, the current dynamic graph has not yet constructed a node corresponding to the XX Braised Chicken product details page. Then, if... Figure 5 As shown, the data acquisition system can construct a node—node 11—corresponding to the product details page of XX Braised Chicken in the current dynamic graph, and create an edge between node 1 and node 11.

[0120] In addition, the data acquisition system can use the Artificial Neural Network (ANN) algorithm to perform vector retrieval of page features in order to calculate the similarity of page features.

[0121] This embodiment does not limit the specific value of the preset similarity threshold, and it can be determined based on requirements, historical records, experiments, etc.

[0122] It's worth noting that the preset similarity threshold doesn't necessarily mean 100% similarity, i.e., complete similarity. The preset similarity threshold can be determined based on the tolerance for visual and / or textual differences within the given scenario.

[0123] For example, such as Figure 7 As shown, both page 1 and page 2 include control 1, control 2, and control 3, with control 3 being the most recently used control. In both page 1 and page 2, control 3 includes control 31, control 32, and control 33.

[0124] On page 1, control 31 is the XX payment control, control 32 is the XX processing control, and control 33 is the video guidance control. On page 2, control 31 is the video guidance control, control 32 is the XX query control, and control 33 is the XX information control.

[0125] In other words, the main difference between page 1 and page 2 lies in control 3—the recently used control. By setting a preset similarity threshold, data collection can identify page 1 and page 2 as the same page, thus avoiding state splits caused by differences in irrelevant parts.

[0126] Based on the above analysis, it can be seen that in this embodiment, the data acquisition system updates the current dynamic graph by adopting a UI state matching mechanism based on image and text multimodal features. This can avoid the repeated construction of the same nodes and the generation of redundant data. It can also avoid the linear growth of the number of nodes due to multiple visits to the same page (such as repeatedly searching for the same keywords), significantly compressing the size of the dynamic graph. Furthermore, it can make the entire exploration process have memory and context awareness capabilities, thereby improving the accuracy and reliability of dynamic graph construction.

[0127] Based on the above analysis, the matching result may indicate that no existing result was matched. However, in other embodiments, the matching result may indicate that an existing result was matched. For example, if there are nodes in the current dynamic graph that have a high similarity to the features of the jump page (e.g., higher than a preset similarity threshold), it means that there is a high probability that a node corresponding to the features of the jump page has already been constructed in the current dynamic graph.

[0128] Accordingly, to avoid the repeated construction of nodes, there is no need to repeatedly construct the same nodes in the current dynamic graph.

[0129] However, in some embodiments, if an existing node is matched, but there is no edge between the existing node and the current node that represents the current operation, the data acquisition system can create a new edge between the existing node and the current node in the dynamic graph, wherein the new edge represents the current operation.

[0130] For example, if the matching result indicates a match with an existing node, the data acquisition system can determine whether there is an edge between the existing node and the current node, and specifically whether there is an edge representing the current operation.

[0131] If not, it means that the path from the current node to the jump page through the current operation has not been recorded. The data acquisition system can complete this transfer relationship, such as creating an edge between the existing node and the current node in the current dynamic graph.

[0132] For example, suppose the homepage includes a "Braised Chicken Bestseller List" control in the recommended section, and this control is not clicked. In a subsequent re-execution of the above steps, the currently unvisited control is the "Braised Chicken Bestseller List" control, and the current action is "click the recommended card".

[0133] Clicking the control redirects to the search results page. This means an existing node—Node 1—has been matched. However, there is no edge between the starting node and Node 1 representing "clicking the recommended card." Therefore, the data collection system does not need to create a new node in the current dynamic graph (to avoid duplication with Node 1), but instead creates an edge between the starting node and Node 1 in the current dynamic graph to represent the "clicking the recommended card" transition relationship from the starting node to Node 1.

[0134] Based on the above analysis, in this embodiment, the data acquisition system, by supplementing missing operation edges even when a state match is successful, avoids independently recording different entry methods (search, recommendation, category, history) of the same page, preventing information loss and ensuring the complete modeling of UI state transition relationships in the dynamic graph. This improves the completeness, comprehensiveness, and reliability of the dynamically generated dynamic graph.

[0135] In other embodiments, if an existing node is matched, and the existing node is the current node, then the control corresponding to the current operation is removed from the control list of the current node.

[0136] For example, if an existing node is matched, the data acquisition system can determine whether the existing node and the current node are the same node. If they are not the same node, the above solution can be used.

[0137] Conversely, if it's the same node, for example, if the UI state doesn't change after the current operation (i.e., the redirected page is the same as the current page), then the data acquisition system can determine that the currently unaccessed control is an invalid control. The data acquisition system can then remove the control corresponding to the current operation from the control list of the current node. Alternatively, the data acquisition system can update the currently unaccessed control to "accessed" or "invalid" to avoid repeated invalid exploration.

[0138] For example, suppose the homepage's control list also includes a "carousel ad" control, which is usually only for display and has no effect when clicked.

[0139] Therefore, if the currently unaccessed control is the "Carousel Ad Image" control, the redirected page will still be the homepage. Thus, the data collection system can remove the "Carousel Ad Image" control from the homepage's control list (or mark it as accessed / invalid).

[0140] In this embodiment, the data acquisition system automatically identifies and removes invalid or non-transfer effect controls by detecting whether the UI state remains unchanged after the operation (i.e., whether the page before and after the current operation is the same page), thereby improving the efficiency of data acquisition.

[0141] Based on the above analysis, in some embodiments, the preset condition can be condition 1) all controls in the control list of each node in the target dynamic graph are accessed controls. In other embodiments, the preset condition can be condition 2) the number of repetitions reaches a preset threshold.

[0142] The preset threshold for the number of attempts can be determined based on requirements, historical records, experiments, and other methods. For example, the number of controls on the homepage and other pages may differ depending on the application scenario. Therefore, a corresponding preset threshold can be determined based on the specific scenario.

[0143] Specifically, if the number of controls on the homepage and other pages of an application in a certain scenario is large, the preset number of times threshold for that scenario may be relatively large; conversely, if the number of controls on the homepage and other pages of an application in a certain scenario is relatively small, the preset number of times threshold for that scenario may be relatively small.

[0144] The repetition count refers to the cumulative number of attempts to perform an operation on the current node (or the entire exploration process) during data acquisition. In this specification, it can be specifically understood as the number of iterations in which operations are performed continuously starting from the starting node but the main termination condition (i.e., all controls have been accessed) is not met.

[0145] For example, a data acquisition system can maintain a counter to determine the number of repetitions.

[0146] For example, after each complete loop of "identify unaccessed controls → perform operation → update dynamic graph" is executed, the data acquisition system increments the counter to record the number of repetitions.

[0147] Correspondingly, after each increment of the counter records the number of repetitions, before executing the next complete loop, the data acquisition system can compare the current number of repetitions with the preset number threshold to determine whether the current number of repetitions has reached the preset number threshold. If it has, the current dynamic graph is determined as the target dynamic graph; otherwise, the next complete loop continues to be executed.

[0148] Additionally, if the preset conditions include condition 1) and condition 2), then if either condition 1) or condition 2) is met, the data acquisition system can stop exploring and use the current dynamic graph as the target dynamic graph.

[0149] The data acquisition system can first determine whether condition 1) is satisfied. If it is, the exploration stops; otherwise, it continues to determine condition 2. Alternatively, the data acquisition system can first determine whether condition 2) is satisfied. If it is, the exploration stops; otherwise, it continues to determine condition 1.

[0150] In this embodiment, the data acquisition system uses a preset threshold number of repetitions as a supplement to the preset conditions, which can effectively cope with challenges such as unpredictable UI, conditionally limited controls, or infinitely scrolling lists, ensuring high coverage while ensuring that the system reliably terminates within a limited time.

[0151] S303: Determine the training data for training the GUI Agent based on the target dynamic graph that meets the preset conditions.

[0152] For example, after constructing the target dynamic graph, the data acquisition system can analyze the target dynamic graph to obtain training data for training the GUI Agent.

[0153] For example, a data acquisition system can use graph traversal or sampling strategies such as depth-first search (DFS), breadth-first search (BFS), and random walk to extract several operation trajectories starting from the starting node from the target dynamic graph, and then generate training data for training the GUI Agent model.

[0154] Based on the analysis of S301 to S303 above, it can be seen that the data acquisition system starts from the application's homepage and dynamically models the transition relationships between UI states during the data acquisition process, constructing and gradually expanding a dynamic graph reflecting the UI interaction structure of the application. Furthermore, during the exploration process of dynamic modeling, it dynamically selects operations to be executed based on unvisited controls on the current page to guide the exploration towards uncovered areas until all controls in the dynamic graph are marked as visited. Based on this dynamic graph, a serialized trajectory containing UI states and corresponding operations is further extracted as training data for the GUI Agent. This approach enables fully automated training data acquisition; it also ensures that the training data closely resembles actual user interaction scenarios, thereby improving the generalization ability and robustness of the trained GUI Agent in real-world environments; it significantly expands the number of training samples, ensuring data sufficiency; it effectively improves the coverage of the state space and the validity of the data; and it makes the process of obtaining training data more efficient and has a wider coverage.

[0155] In some embodiments, S303 may include: extracting a number of operation trajectories corresponding to the application from the target dynamic graph, and determining the intent corresponding to each operation trajectory to obtain training data including a number of training pairs, wherein a training pair includes an operation trajectory and a corresponding intent.

[0156] Operation trajectory refers to the ordered sequence of states and actions formed by traversing several nodes along the edges from the starting node in the target dynamic graph.

[0157] An intent refers to the user's goal or task semantics that drives the completion of a corresponding action trajectory. It is usually represented by natural language phrases or structured task labels, such as "order a braised chicken rice to be delivered to the company" in the example above. In scenarios where intents are available, they can be used as supervisory signals to train a GUI agent to implement the mapping between "intent" and "action".

[0158] Training pairs are the basic building blocks of training data. A training pair consists of a sample that includes an operational trajectory and its corresponding intention.

[0159] For example, the data acquisition system can traverse all valid paths in the target dynamic graph, such as starting from the starting node and ending at a node with a clear business endpoint, such as "Confirm Payment Page" or "Order Success Page".

[0160] For each of the aforementioned valid paths, the data acquisition system can extract nodes (UI states) and edges (operations) in chronological order to reconstruct the complete operation trajectory.

[0161] For each valid path, the data acquisition system can determine the user intent (user purpose) of that valid path, and thus construct a training pair with that valid path. The training pair includes the operation trajectory and corresponding intent corresponding to that valid path.

[0162] Continuing with the example above, let's assume the target animation has covered the valid path: Homepage → Search "Braised Chicken Rice" → Select Merchant → Add Product → Proceed to Checkout → Select Address "Company" → Pay Now.

[0163] Therefore, the data acquisition system can extract the corresponding operation trajectory from the target dynamic graph: Homepage, click the search box and enter "braised chicken rice" → search results page, click "XX braised chicken" → product details page, click "+" → shopping cart page, click "go to checkout" → order confirmation page, select address "company" → payment page, click "pay now".

[0164] Continuing with the example above, this operation trajectory is constructed in response to the user's voice command "Order a braised chicken rice and have it delivered to the company," therefore, its intent is "Order a braised chicken rice and have it delivered to the company."

[0165] Thus, the data acquisition system can obtain training pairs that include the operation trajectory and the corresponding intention.

[0166] In other embodiments, after the data acquisition system extracts each operation trajectory, it can predict the intent corresponding to each operation trajectory through a pre-trained intent prediction big language model, thereby obtaining training data including several training pairs.

[0167] Similarly, this specification does not limit the training method for the intent prediction large language model. For example, the data used to train the intent prediction large language model is sample data, which may include operation trajectory samples and intent labels. The intent prediction large language model is trained by minimizing the difference between the predicted intent and the intent label.

[0168] Based on the above analysis, it can be seen that in this embodiment, the data acquisition system extracts the operation trajectory from the target dynamic image and binds it to the user's intent, thereby transforming the results of automated exploration into training data, which can improve the accuracy and reliability of the training data.

[0169] In some embodiments, after extracting several operation trajectories corresponding to the application from the target dynamic graph, the data acquisition system can perform a quality assessment on the several operation trajectories based on preset quality assessment indicators to obtain operation trajectories that pass the quality assessment. The preset quality assessment indicators include length and / or validity.

[0170] Accordingly, the data acquisition system can determine training data based on operation trajectories that have passed quality assessment. For example, for each operation trajectory that has passed quality assessment, the data acquisition system can determine the intent corresponding to that operation trajectory to obtain the corresponding training pair.

[0171] Preset quality assessment metrics are calculable standards used to quantify the quality of operational trajectories, in order to select high-quality samples that are truly valuable for GUIAgent training.

[0172] Preset quality assessment metrics can include length metrics. Length refers to the number of operation steps (such as the number of UI state transitions) contained in the operation trajectory.

[0173] For example, a short operation path may indicate that the task is not completed (such as simply clicking the homepage), while a long operation path may contain redundant or cyclical operations.

[0174] Therefore, in some embodiments, when the data acquisition system performs quality assessment on the corresponding operation trajectory based on the length index, if the length of the operation trajectory is between a first preset length threshold and a second preset length threshold, then the operation trajectory is characterized as an operation trajectory that has passed the quality assessment.

[0175] The first preset length threshold represents a relatively short length to avoid operation trajectories that are too short (e.g., the corresponding task is not completed); the second preset length threshold represents a relatively long length to avoid operation trajectories containing redundant or cyclical operations. Similarly, this embodiment does not limit the specific values ​​of the first and second preset length thresholds, and they can be determined based on requirements, historical records, experiments, etc.

[0176] Validity refers to whether the operation trajectory successfully achieves a meaningful task objective, such as whether a key endpoint page (e.g., "payment success page," "order confirmation page") has been reached. Relatively speaking, a valid trajectory should reflect a complete user task loop.

[0177] It is worth noting that in some approaches, the data acquisition system can evaluate the quality of the operation trajectory based on length; in others, it can evaluate the quality based on validity; and in still others, it can evaluate the quality based on both length and validity.

[0178] For example, a data acquisition system can evaluate the quality of an operation trajectory based on its length and validity: The data acquisition system can first evaluate the quality of the operation trajectory based on its length. If the operation trajectory passes the length quality evaluation, it can then continue to evaluate the quality of the operation trajectory based on its validity. Conversely, if the operation trajectory fails the length quality evaluation, there is no need to perform a validity quality evaluation.

[0179] Similarly, the data acquisition system can first evaluate the quality of the operation trajectory based on validity. If the operation trajectory passes the validity quality evaluation, then the quality of the operation trajectory can be evaluated based on length. Conversely, if the operation trajectory fails the validity quality evaluation, then there is no need to evaluate the length quality.

[0180] Based on the above analysis, it can be seen that in this embodiment, the data acquisition system can eliminate invalid and fragmented operation trajectories that do not meet quality requirements through a trajectory quality evaluation mechanism based on length and / or validity, thereby improving the accuracy and reliability of training data.

[0181] In some embodiments, after extracting several operation trajectories corresponding to the application from the target dynamic graph, the data acquisition system can filter the several operation trajectories.

[0182] For example, the data acquisition system can filter several operation trajectories based on preset filtering rules. The preset filtering rules are used to identify and remove operation trajectories that contain risky pages. The risky pages are pages that include preset sensitive keywords and / or sensitive control types.

[0183] Preset filtering rules refer to the judgment logic used to identify and exclude high-risk or non-compliant operation trajectories. In this embodiment, preset filtering rules can be determined based on page content features.

[0184] Risk pages refer to pages that may lead to privacy breaches, security violations, legal risks, or training biases.

[0185] The identification criteria for risky pages include: preset sensitive keywords and / or sensitive control types. Preset sensitive keywords refer to words containing sensitive information, such as "password," "ID card," "bank card," and "real-name authentication." Sensitive control types refer to controls containing sensitive information, such as "password input box," "file upload control," and "permission request pop-up."

[0186] It is understandable that the specific content on the risk page can be determined based on the application area, compliance requirements, training objectives, etc. The above example is only used to illustrate the possible content of the risk page and should not be construed as a limitation on the risk page.

[0187] Taking preset sensitive keywords as an example: Risk pages can correspond to a set of sensitive keywords. For each operation trajectory, the data acquisition system can traverse each page in that trajectory based on this set. If a page contains a sensitive keyword from this set, the data acquisition system can identify that page as a risk page and filter out the operation trajectory to avoid including it in the training data.

[0188] Taking sensitive control types as an example: Risk pages can correspond to a set of sensitive control types. For each operation trajectory, the data acquisition system can traverse each page in that trajectory based on this set. If a page contains a sensitive control type included in the set, the data acquisition system can identify that page as a risk page and filter out the operation trajectory to avoid including it in the training data.

[0189] If a risky page contains both preset sensitive keywords and sensitive control types, the data collection system can identify it as a risky page. Alternatively, the data collection system can identify a page that meets either of these criteria as a risky page.

[0190] Based on the above examples, in this embodiment, the data acquisition system can prevent the leakage of sensitive information and thus improve data security by filtering several operation trajectories based on preset sensitive keywords and / or sensitive control types.

[0191] Furthermore, if both quality assessment and screening are included, this embodiment does not limit the order of quality assessment and screening. For example, quality assessment can be performed first to screen the operation trajectories that pass the quality assessment; alternatively, screening can be performed first to perform quality assessment on the remaining operation trajectories.

[0192] It is worth noting that the above examples are merely illustrative of possible implementations of the data acquisition method in this specification, and should not be construed as limiting the implementation of the data acquisition method in this specification. For example, based on the above technical concept, some of the technical features described above can be combined to obtain new embodiments; new technical features can be added to the above examples to obtain new embodiments; some technical features can be removed from the above examples to obtain new embodiments; some technical features in the above examples can be replaced with other technical features; some technical features and their order in the above examples can be adjusted to obtain new embodiments, and so on, which will not be listed here.

[0193] Based on the above technical concept, this specification also provides a computer-readable non-transitory storage medium storing at least one instruction set, which, when executed by a processor, implements the steps of the data acquisition method described in this specification.

[0194] In some possible implementations, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on the data acquisition system 200, the program code causes the data acquisition system 200 to perform the steps of the data acquisition method described in this specification. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the data acquisition system 200. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the data acquisition system 200, partially on the data acquisition system 200, as a standalone software package, partially on the data acquisition system 200 and partially on a remote data acquisition system, or entirely on the remote data acquisition system 200.

[0195] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of user-related information in the technical solutions of this specification all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0196] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0197] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0198] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0199] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and to aid in understanding a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art, upon reading this specification, may readily identify some of the devices as separate embodiments. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. And the content of each secondary embodiment is valid even if it contains fewer than all the features of a single foregoing disclosed embodiment.

[0200] Every patent, patent application, publication of a patent application, and other material cited herein, such as articles, books, specifications, publications, documents, and literature (excluding any related historical examination documents), is referenced for all purposes relevant to this document, including in the specification and claims herein. However, in the event of any inconsistency or conflict between the descriptions, definitions, and / or terms used in the foregoing and those used herein, the descriptions, definitions, and / or terms used herein shall prevail.

[0201] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A data acquisition method, comprising: An initialization animation of the application is provided, wherein the initialization animation includes a starting node corresponding to the application's homepage, and the starting node includes a screenshot of the homepage and a list of controls; Repeat the following steps until a target animated image that meets the preset conditions is obtained, wherein the preset conditions include that all controls in the control list of each node in the target animated image are accessed controls: Identify currently unaccessed controls from the control list of the current node; Determine and execute the current operation corresponding to the currently unaccessed control to obtain the redirected page; The current dynamic graph is updated according to the redirected page, wherein the updated dynamic graph includes a first node corresponding to the redirected page and a current edge between the current node and the first node. The first node includes a screenshot of the redirected page and a list of controls, and the current edge represents the current operation. Update the currently unaccessed controls based on the unaccessed controls in the control list of the first node; Training data for training the GUI Agent is determined based on the target dynamic graph that meets the preset conditions.

2. The method according to claim 1, wherein, If there are no unaccessed controls in the control list of the first node, then the currently unaccessed controls are updated according to the unaccessed controls in the control list of the parent node of the first node.

3. The method according to claim 1, wherein, Identify currently unaccessed controls from the control list of the current node, including: Determine the set of reachable pages after executing the unaccessed controls in the control list of the current node; For each reachable page in the reachable page set, determine the number of unaccessed controls on that reachable page, and identify the unaccessed control in the control list of the current node with the highest number of unaccessed controls as the currently unaccessed control.

4. The method according to claim 1, wherein, Each node in the current dynamic graph also includes the page features of the page corresponding to that node; Update the current animated image based on the redirected page, including: Obtain the UI state of the redirected page, which includes a screenshot of the redirected page and a list of controls; Based on the UI state of the redirected page, the redirected page features are extracted, including the image features and text features of the redirected page; Match the jump page features with the page features of each node in the current dynamic graph; and If no existing node is matched, the first node is created in the current dynamic graph based on the UI state of the redirected page, and the current edge is also created.

5. The method according to claim 4, wherein, The method further includes: If an existing node is matched, but there is no edge between the existing node and the current node that represents the current operation, then a new edge is created between the existing node and the current node in the dynamic graph, wherein the new edge represents the current operation.

6. The method according to claim 4, wherein, The method further includes: If an existing node is matched, and the existing node is the current node, then the control corresponding to the current operation is removed from the control list of the current node.

7. The method according to claim 1, wherein, The preset conditions also include the number of repetitions reaching a preset threshold.

8. The method according to claim 1, wherein, Training data for training the GUI Agent is determined based on the target dynamic graph that meets the preset conditions, including: Extract several operation trajectories corresponding to the application from the target dynamic graph; and For each operation trajectory, the corresponding intent is determined to obtain training data including several training pairs, wherein a training pair includes an operation trajectory and a corresponding intent.

9. The method according to claim 8, wherein, The method further includes: The quality of the several operation trajectories is evaluated based on preset quality evaluation indicators to obtain operation trajectories that pass the quality evaluation. The preset quality evaluation indicators include length and / or validity.

10. The method according to claim 8, wherein, The method further includes: The operation trajectories are filtered based on preset filtering rules, wherein the preset filtering rules are used to identify and remove operation trajectories containing risky pages, wherein the risky pages are pages that include preset sensitive keywords and / or sensitive control types.

11. The method according to claim 1, wherein, If the repeated steps are being performed for the first time, the currently unaccessed control is the unaccessed control determined from the control list on the homepage.

12. The method according to claim 1, wherein, The application is one of the following: e-commerce application, social application, content information application, or utility application.

13. A data acquisition system, comprising: At least one storage medium storing at least one instruction set for data acquisition; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, it reads the at least one instruction set and executes the method as described in any one of claims 1 to 12 according to the instructions of the at least one instruction set.

Images