Method and device for managing user operation data

Abstract

The embodiment of the invention provides a method and device for managing user operation data, and the method for managing the user operation data comprises the steps: obtaining a plurality of operation tracks; wherein any first operation track is obtained by operating the first application according to the target agent aiming at the first task, and the first operation track comprises page information of a plurality of pages arranged in sequence and operation needing to be executed for skipping of every two adjacent pages. A relation graph is constructed according to the multiple operation tracks, the relation graph comprises multiple nodes and connecting edges among the nodes, each node corresponds to one page in the multiple pages operated by the multiple operation tracks, and the connecting edges correspond to operation between two adjacent pages included in the multiple operation tracks.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of data processing technology, and in particular relate to a method and apparatus for managing user operation data. Background Technology

[0002] With the rapid deployment of large-scale models and agent technologies in the field of graphical user interface (GUI) automation on mobile phones, computers, and mini-programs, GUI-based agents (hereinafter referred to as GUI agents) are now able to replace humans in performing various tasks, such as "online appointment registration" and "self-service coffee ordering." During the execution of a task by a GUI agent, the system can record the specific steps required to complete the task, while capturing screenshots and other page information for each step, as well as the corresponding GUI operations (such as clicking, inputting, and swiping). All these steps, combined sequentially, constitute a GUI operation trajectory data. This type of data can be further used to train and optimize the GUI agent, improving its task execution capabilities.

[0003] However, in traditional technical solutions, GUI operation trajectory data is stored in the form of a linear sequence, which has limitations in terms of flexibility in combining, expanding, and synthesizing new training samples. In addition, when the software or app is updated (e.g., interface redesign or function adjustment), the original operation trajectory data often cannot be partially repaired or rolled back, resulting in a large amount of historical data being discarded and needing to be re-collected, causing a waste of efficiency and resources.

[0004] Therefore, there is an urgent need to build a reusable and easy-to-maintain user operation data management solution to support GUI agents in continuously learning and adapting in dynamically changing interface environments. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for managing user operation data. A unified relationship diagram is constructed based on multiple operation trajectories, and centralized management is performed on this diagram. This can significantly improve the utilization efficiency of operation data and greatly enhance the maintainability of operation data.

[0006] The first aspect of this specification provides a method for managing user operation data, including:

[0007] Multiple operation trajectories are obtained; wherein any first operation trajectory is obtained based on the target intelligent agent's operation on the first application for the first task, and the first operation trajectory includes page information of multiple pages arranged in sequence and the operation to be performed for each two adjacent pages;

[0008] A relationship graph is constructed based on the multiple operation trajectories. The relationship graph includes multiple nodes and connecting edges between nodes. Each node corresponds to one of the multiple pages operated by the multiple operation trajectories, and the connecting edge corresponds to the operation between two adjacent pages included in the multiple operation trajectories.

[0009] A second aspect of this specification provides an apparatus for managing user operation data, comprising:

[0010] An acquisition unit is used to acquire multiple operation trajectories; wherein any first operation trajectory is obtained based on the target intelligent agent's operation on the first application for the first task, and the first operation trajectory includes page information of multiple pages arranged in sequence and the operation to be performed for each jump between two adjacent pages;

[0011] A construction unit is configured to construct a relationship graph based on the multiple operation trajectories. The relationship graph includes multiple nodes and connecting edges between nodes. Each node corresponds to one of the multiple pages operated by the multiple operation trajectories, and the connecting edge corresponds to the operation between two adjacent pages included in the multiple operation trajectories.

[0012] A third aspect of this specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0013] A fourth aspect of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in the first aspect.

[0014] This specification provides a computer program product in a fifth aspect, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0015] The method and apparatus for managing user operation data provided in one or more embodiments of this specification form a relationship graph by constructing the pages operated by multiple operation trajectories as nodes and constructing the operations between two adjacent pages as connecting edges, and then centrally managing the data on this graph. This can significantly improve the efficiency of data utilization and greatly enhance the maintainability of the data. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;

[0018] Figure 2 A schematic diagram of a system for managing user operation data is shown in one example of this specification.

[0019] Figure 3 This diagram illustrates a method for managing user operation data according to one embodiment of the present specification.

[0020] Figure 4 This document illustrates an example of the operational trajectory shown in this specification.

[0021] Figure 5 This document shows a schematic diagram of the operating trajectory in another example from this specification;

[0022] Figure 6 A schematic diagram of relationships is shown in one example of this specification;

[0023] Figure 7 A schematic diagram of relationships is shown in another example in this specification;

[0024] Figure 8 A schematic diagram illustrating the answer path in one example of this specification;

[0025] Figure 9 A schematic diagram of an apparatus for managing user operation data according to one embodiment of this specification is shown. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0027] As mentioned earlier, linear storage operation trajectories suffer from the problems of non-reusability and difficulty in maintenance. To address these issues, the following improvement schemes are proposed:

[0028] First, the automation scripts from the Robotic Process Automation (RPA) platform are saved. However, this approach has the following drawbacks: the scripts are usually strongly coupled with the specific environment (coordinates, window position), resulting in poor portability; and it is difficult to synthesize new trajectories based on existing ones.

[0029] Second, it provides page screenshots and UI element trees. This approach is suitable for interface understanding research, but it lacks connectivity of action edges / cross-page causal relationships and cannot be directly used to train agents for cross-page navigation.

[0030] Third, the operation trajectory can be reconstructed using system events or accessibility logs. However, this approach has the following drawbacks: the data is noisy, lacks semantic meaning (i.e., which element represents what business intent), and is usually still stored in a linear sequence, lacking merging and reuse mechanisms.

[0031] Fourth, it can be stored as an interactive diagram or a webpage navigation diagram. However, this approach has the following drawbacks: it is only applicable to diagram representations of webpages or specific scenarios, and lacks versatility.

[0032] Given the shortcomings of the above-mentioned solutions, this solution proposes to construct a relationship graph by building the pages operated by multiple operation trajectories into nodes and the operations between two adjacent pages into connecting edges. Centralized management can be carried out on this graph, which can significantly improve the efficiency of data utilization and greatly enhance the maintainability of data.

[0033] Figure 1 This is a schematic diagram illustrating an implementation scenario of one of the embodiments disclosed in this specification. Figure 1 In this system, multiple operation trajectories can be obtained: S1, S2, ..., Sn. A single operation trajectory can include page information of multiple sequentially arranged pages and the operation required to jump between every two adjacent pages. The page information can include page screenshots or page names. Taking page screenshots as an example, operation trajectory S1 can be represented as: <Page Screenshot 11, Operation 11, Page Screenshot 12, ...>. Operation trajectory S2 can be represented as: <Page Screenshot 21, Operation 21, Page Screenshot 22, ...>, and so on. Operation 11 or Operation 21 belongs to the following operation categories: click operation, swipe operation, long press operation, input operation, or selection operation.

[0034] Then, a relationship graph can be constructed based on multiple operation trajectories, where each node in the graph corresponds to a page and each connecting edge corresponds to an operation.

[0035] Each node in the above relationship graph has an image representation and a text representation. The image representation can be obtained by embedding a screenshot of the page corresponding to the node at a certain moment; the text representation can be obtained by embedding the path information (referred to as the page path) of the page corresponding to the node in the DOM tree.

[0036] Finally, the multiple nodes and the connecting edges between them can be stored. Furthermore, for each node, its image representation and text representation can also be stored in association.

[0037] The method for managing user operation data described in this solution can be executed by a system for managing user operation data, wherein the system for managing user operation data can be as follows: Figure 2 As shown, Figure 2 In this system, the following modules may be included:

[0038] The data collection module can be implemented on mobile devices, test machines, browser plugins, or mini-program sandboxes, and is used to collect multiple operation trajectories. A single operation trajectory may include page screenshots, page paths, input parameters, environmental context, and manual annotations.

[0039] The data access module is used to perform noise reduction, image and text consistency comparison, operation time correction, and metadata completion on the operation trajectory.

[0040] The state normalization module is used for similarity calculation and merging, as well as manual verification of low confidence levels.

[0041] The edge construction and semantic markup module is used to construct connecting edges between nodes, record operation metadata, calculate edge weights, and perform statistics.

[0042] The graph storage and indexing module stores information such as nodes, connecting edges, edge attributes, and versions. It also stores embedding vectors and label indexes.

[0043] The query and synthesis module is used to query the shortest, optimal, or constrained paths, and to synthesize new trajectories.

[0044] The playback and verification module is used to replay the synthesized new trajectory on a real device / emulator, perform reachability checks at each step, and collect reasons for failures.

[0045] The management and decommissioning module is used for version control, decommissioning / redirecting, and manual review / commenting interfaces.

[0046] This solution avoids data duplication and redundancy when managing user operation data using the aforementioned system, and supports more efficient data synthesis schemes, faster data update mechanisms, and more flexible data application capabilities.

[0047] Figure 3 This diagram illustrates a method for managing user operation data according to one embodiment of this specification. This method can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. Figure 3 As shown, the method may include the following steps:

[0048] S302, acquire multiple operation trajectories.

[0049] In one embodiment, the operation trajectory can be obtained from the GUI interaction log. The GUI interaction log is a detailed record file generated in real time by a GUI agent (or any automated testing tool, RPA robot) during the operation of an application for a specific task.

[0050] First, the aforementioned GUI agent can run on mobile devices, test machines, browser plugins, or mini-program sandboxes.

[0051] In one example, the GUI interaction log mentioned above may include: operation (e.g., click operation, swipe operation, etc.), operation time, operation element, input parameters (e.g., what text was entered in the input box, which option was selected from the drop-down menu), operation status (success or failure), page screenshot, page path, etc.

[0052] Specifically, page screenshots and / or page paths can be collected sequentially from the GUI interaction logs, and the operations required to jump between the two pages corresponding to each pair of adjacent page screenshots (or page paths) can be collected to obtain the corresponding operation trajectory.

[0053] Of course, in practice, in addition to collecting page screenshots and / or page paths and operations, you can also collect page information such as environmental context (e.g., page login status, network information) to form an operation trajectory. This manual does not limit this.

[0054] It should be understood that the aforementioned multiple operation trajectories can be obtained by collecting multiple GUI interaction logs generated by the GUI agent operating different applications for multiple tasks.

[0055] In another embodiment, the above-mentioned operation trajectory can also be obtained by collecting manually labeled trajectories. Here, the manually labeled trajectory includes multiple sequentially executed steps, each step including page information for a page and the operation performed on that page.

[0056] Figure 4 The diagram shows an example of an operation trajectory in this specification. This operation trajectory can be obtained by collecting GUI interaction logs generated by the GUI agent during the operation of Alipay for the task: "Use Zhejiang Government Service Platform to register for an orthopedic appointment in Alipay". Figure 4 The operation trajectory includes screenshots of the "Alipay search page", the "Zhejiang Government Service Platform homepage", the "Health and Medical Insurance Page", and the "Appointment Registration Page", as well as the "Zhejiang Government Service Platform" input operation when jumping from the "Alipay search page" to the "Zhejiang Government Service Platform homepage", the "Health and Medical Insurance Page" click operation when jumping from the "Zhejiang Government Service Platform homepage" to the "Health and Medical Insurance Page", and the "Appointment Registration Page" click operation when going from the "Health and Medical Insurance Page" to the "Appointment Registration Page".

[0057] Figure 5 The diagram shows an operation trajectory in another example of this specification. This operation trajectory can be obtained by collecting GUI interaction logs generated by the GUI agent during the operation of the micro-service in response to the task: "Check social security using Zhejiang government service in micro-service". Figure 5 The operation trajectory includes screenshots of the "WeChat Search Page", the "Zhejiang Government Service Platform Homepage", the "Social Security Page", and the "Social Security Inquiry Page", as well as the "Zhejiang Government Service Platform" input operation when jumping from the "WeChat Search Page" to the "Zhejiang Government Service Platform Homepage", the "Social Security" click operation when jumping from the "Zhejiang Government Service Platform Homepage" to the "Social Security Page", and the "Social Security" click operation when going from the "Social Security Page" to the "Social Security Inquiry Page".

[0058] Additionally, the collected operation trajectories can be cleaned (e.g., noise reduction, image-text consistency comparison, and operation time correction). Furthermore, metadata can be added, which may include, but is not limited to, application identifiers, the system language of the acquisition device, and hardware information.

[0059] S304, construct a relationship graph based on multiple operation trajectories. This relationship graph includes multiple nodes and connecting edges between nodes.

[0060] Typically, there will be overlapping information among the multiple operation trajectories obtained above, for example, Figure 3 The operation trajectory shown and Figure 4 The screenshots of the second page included in the shown operation paths are essentially identical. Therefore, to improve management efficiency, the duplicate information in these two operation paths can be merged.

[0061] Taking the page information mentioned above as a screenshot as an example, for any two operation trajectories, the similarity between the page screenshots included in the two operation trajectories is calculated. Similar page screenshots are selected from the two operation trajectories and merged to obtain a merged screenshot. This merged screenshot is then used to update the two operation trajectories, resulting in two updated trajectories. A relationship graph is then constructed based on the updated trajectories determined for multiple operation trajectories.

[0062] Assume the two operation trajectories mentioned above are S1 and S2, and operation trajectory S1 can be represented as: <page screenshot 11, operation 11, page screenshot 12, operation 12, page screenshot 13, operation 13 and page screenshot 14>, and operation trajectory S2 can be represented as: <page screenshot 21, operation 21, page screenshot 22, operation 22, page screenshot 23, operation 23 and page screenshot 24>. Then, we can calculate the similarity between page screenshot 11 and page screenshots 21, 22, 23, and 24, respectively; calculate the similarity between page screenshot 12 and page screenshots 21, 22, 23, and 24, respectively; calculate the similarity between page screenshot 13 and page screenshots 21, 22, 23, and 24, respectively; and calculate the similarity between page screenshot 14 and page screenshots 21, 22, 23, and 24, respectively.

[0063] In a more specific embodiment, for any page screenshot, an image encoder can be used to embed the screenshot to obtain a corresponding image representation. This image encoder can employ a general image classification network, such as convolutional-based ResNet, VGG, DenseNet, etc., or Transformer-based ViT, T2T-ViT, etc. Then, based on this image representation, the similarity between two page screenshots is calculated.

[0064] In another, more specific embodiment, for any page screenshot, a text encoder can be used to embed the page path of the screenshot (i.e., the path of the page corresponding to the screenshot in the DOM tree) to obtain the corresponding text representation. Here, the text encoder can employ a Transformer-based network, such as BERT or RoBERTa, or an RNN-based model such as LSTM or Bi-LSTM. Then, based on this text representation, the similarity between two page screenshots is calculated.

[0065] After calculating the similarity between each pair of page screenshots, if the similarity between any two page screenshots is greater than a predetermined threshold, then the two page screenshots are determined to be similar page screenshots. Then, either of the two page screenshots can be used as a merged screenshot, and this merged screenshot can be used to replace the other page screenshot, thereby obtaining the updated trajectory.

[0066] Taking the two operation trajectories S1 and S2 mentioned above as examples, assuming that page screenshot 12 and page screenshot 22 are similar page screenshots, and assuming that page screenshot 12 is taken as the merged screenshot, then we can get two update trajectories: S1′=S1=<page screenshot 11, operation 11, page screenshot 12, operation 12, page screenshot 13, operation 13 and page screenshot 14>; S2′=<page screenshot 21, operation 21, page screenshot 12, operation 22, page screenshot 23, operation 23 and page screenshot 24>.

[0067] In summary, this solution merges similar page screenshots from multiple operation trajectories, which can significantly improve data utilization and reduce collection and maintenance costs.

[0068] Of course, in practice, a relationship diagram can also be constructed directly based on multiple operation trajectories, and this manual does not limit this.

[0069] It should be noted that the nodes in the relationship graph constructed in this scheme can correspond to one of the multiple pages operated by multiple operation trajectories (or multiple update trajectories), and the connecting edges correspond to the operations between two adjacent pages included in multiple operation trajectories.

[0070] Again, based on the above Figure 4 and Figure 5 Taking the two operation trajectories in the example, Figure 4 A screenshot of the "Zhejiang Government Service Platform Homepage" and Figure 5 The screenshots of the "Zhejiang Government Service Platform Homepage" are similar to those of other pages, so they can be merged. One of them can be used as the merged screenshot, and this merged screenshot can be used to update both operation paths. Finally, the relationship diagram built based on these two update paths can be as follows: Figure 6 As shown.

[0071] Figure 6In this diagram, the relationship graph is a directed graph containing 7 nodes. These 7 nodes correspond to the following pages: "Alipay Search Page," "Zhejiang Government Service Website," "Health and Medical Insurance Page," "Appointment Registration Page," "WeChat Search Page," "Social Security Page," and "Social Security Inquiry Page." The connection between "Alipay Search Page" and "Zhejiang Government Service Website" corresponds to the input of "Zhejiang Government Service," as does the connection between "WeChat Search Page" and "Zhejiang Government Service Website." The two connections between "Zhejiang Government Service Website" and "Health and Medical Insurance Page" correspond to clicking "Health and Medical Insurance" and clicking "Back," respectively; the two connections between "Health and Medical Insurance Page" and "Appointment Registration Page" correspond to clicking "Appointment Registration" and clicking "Back," respectively; the two connections between "Zhejiang Government Service Website" and "Social Security Page" correspond to clicking "Social Security" and clicking "Back," respectively; and the two connections between "Social Security Page" and "Social Security Inquiry Page" correspond to clicking "Social Security" and clicking "Back," respectively.

[0072] It should be noted that this solution constructs a unified relationship diagram based on multiple operation trajectories and centrally manages them on this diagram, which makes user operation data traceable and auditable, and facilitates compliance and manual inspection.

[0073] After constructing the aforementioned relationship graph, for each node, an image encoder can be used to embed a screenshot of the page corresponding to that node to obtain its image representation. Furthermore, for each node, a text encoder can be used to embed the page path of the page corresponding to that node to obtain its text representation.

[0074] Then, the above relationship graph can be stored. For example, multiple nodes and the connecting edges between nodes can be stored in a graph database. For a single node, its node attributes can also be stored, which may include one or more of the following: page path, application identifier, environment context, and metadata. Similarly, for a single connecting edge, its edge attributes can be stored, which may include one or more of the following: operation element (e.g., Alipay's "Scan" or "Payment Code"), execution time, operation status (success or failure), operation position (horizontal and vertical coordinates), input parameters, and termination node.

[0075] Of course, in practice, the corresponding edge weight can also be associated and stored for each connection edge.

[0076] In one example, for any connecting edge, the corresponding edge weight can be calculated based on one or more of the following: the execution frequency of the corresponding operation, the success rate of the subsequent path, and the execution time of the corresponding operation.

[0077] Finally, it can also store information such as the version and source of nodes / connections.

[0078] For the stored relationship graph, when the application interface or functionality changes, the target nodes and / or target connecting edges in the graph can be updated. This updating includes adding or modifying elements. Using a real device or simulator, the operations related to the target node and / or the operations corresponding to the target connecting edge are replayed and verified, including determining whether the execution result of the operation is correct. If the replay verification fails, the corresponding modifications are executed.

[0079] It should be noted that other nodes and connecting edges in the above relationship graph can be automatically reused to avoid manual modification of the entire linear trajectory.

[0080] by Figure 6 Taking the relationship diagram shown as an example, a connecting edge can be added between the "Alipay Search Page" and the "Health Insurance Page". The relationship diagram after adding the connecting edge can be as follows: Figure 7 As shown.

[0081] In addition, version control, offline marking (i.e. deletion) can be applied to nodes or edges in the relationship graph, migration suggestions (e.g., redirecting old nodes to new nodes) can be provided, and a manual review / annotation interface can be added (e.g. merging conflicts, correcting semantic tags, etc.) can be implemented.

[0082] It should also be noted that in this scheme, for a single node, the image representation and text representation of the node can be stored in the vector database, and a corresponding vector index or label index can be built for the vector database for querying.

[0083] The relationship graph constructed above can be used to build training samples for the GUI agent. Furthermore, it can be configured into the GUI agent so that the agent can quickly execute new tasks based on the relationship graph.

[0084] The following section will first explain the process of constructing training samples for GUI agents based on this relationship graph.

[0085] Specifically, a target trajectory can be synthesized based on the constructed relationship graph. Using real equipment or a simulator, the various target operations included in the target trajectory are replayed and verified. The replay verification for a single target operation includes determining whether the execution result of that target operation is correct. After successful replay verification, the target trajectory is used as a training sample for training the GUI agent.

[0086] First, the target trajectory mentioned above can be obtained by combining different incoming and outgoing edges of a certain node (hereinafter referred to as the target node).

[0087] In one example, a target trajectory can be synthesized based on the pre-order path a and the post-order path b of the target node, where the pages covered by the pre-order path a and the pages covered by the post-order path b belong to different operation trajectories.

[0088] Taking Figure 6 the relational graph shown as an example, the node corresponding to the "Zheli Office Home Page" can be used as the target node, and the pre-order path a of the target node: "Alipay Search Page" -> "Zheli Office Home Page" and the post-order path b of the target node: "Zheli Office Home Page" -> "Social Security Page" -> "Social Security Inquiry Page" can be combined to form the target trajectory S3. Because the pages covered by the pre-order path a: "Alipay Search Page" and "Zheli Office Home Page" and the pages covered by the post-order path b: "Zheli Office Home Page", "Social Security Page" and "Social Security Inquiry Page" belong to different operation trajectories.

[0089] In addition, the pre-order path c of the target node: "WeChat Search Page" -> "Zheli Office Home Page" and the post-order path d of the target node: "Zheli Office Home Page" -> "Healthcare Insurance Page" -> "Reservation Registration Page" can be combined to form the target trajectory S4. Because the pages covered by the pre-order path c: "WeChat Search Page" and "Zheli Office Home Page" and the pages covered by the post-order path d: "Zheli Office Home Page", "Healthcare Insurance Page" and "Reservation Registration Page" belong to different operation trajectories.

[0090] Secondly, the playback verification (also known as reachability verification) of each target operation included in the target trajectory is actually to judge whether the execution results of each target operation are correct. Taking the above target trajectory S3 as an example, it can be judged whether it can jump to the "Zheli Office Home Page" after entering "Zheli Office" on the "Alipay Search Page". And judge whether it can jump to the "Social Security Page" after clicking "Social Security" on the "Zheli Office Home Page". It is also judged whether it can jump to the "Social Security Inquiry Page" after clicking "Social Security" on the "Social Security Page", and so on.

[0091] Of course, in practice, the playback verification can also be combined with the operation state, and this specification does not limit this.

[0092] It should be understood that when the playback verification of the above target trajectory fails, the failure reasons (such as page changes, element non-existence, permission issues, etc.) can be collected and analyzed to update the relational graph. For example, when the verification of a certain operation fails, the execution success rate of the connection edge corresponding to this operation can be updated, new nodes / connection edges can be added, or it can be marked as offline, etc.

[0093] Finally, for the obtained target trajectory, a function description can be added to it, and thus a training sample for the GUI agent is obtained.

[0094] It should be noted that this solution generates training samples for GUI agents based on relationship graphs, which can greatly expand the training samples and thus reduce the cost of manual collection.

[0095] The above describes the process of constructing training samples for GUI agents. The following describes the process of GUI agents quickly executing new tasks based on this relationship graph.

[0096] Specifically, a target prompt can be input into the GUI agent. This target prompt is used to indicate the first and last nodes in the relationship graph for the user question. The target path is determined based on the edge weights or execution times of the connecting edges between the first and last nodes. The answer to the user question is then output based on the target path.

[0097] by Figure 6 Taking the relationship diagram shown as an example, suppose the user's question is: "I'm currently on the social security page, what's the fastest way to make an appointment?", then the GUI agent's output could be: Click "Back" on the "Social Security Page" to jump to the "Zhejiang Government Service Homepage"; click "Health Insurance" on the "Zhejiang Government Service Homepage" to jump to the "Health Insurance Page"; click "Appointment Registration" on the "Health Insurance Page" to jump to the "Appointment Registration Page". The path corresponding to this answer (referred to as the answer path) can be found in [link to answer path diagram]. Figure 8 As shown.

[0098] It should be noted that the above-mentioned GUI agent generates answers to user questions based on relationship graphs, which can greatly improve the execution efficiency and reliability of the GUI agent.

[0099] In summary, this solution has the following innovative features: 1. Graph-structured operation trajectory storage: Pages are abstracted as nodes in a graph, and interactions between pages (clicks, inputs, swipes, etc.) are abstracted as connecting edges, constructing a queryable and combinable relationship graph. Combining a graph database and vector index enables large-scale storage and efficient retrieval by representation / tag, and supports versioning, decommissioning, and redirection of nodes / connecting edges, thereby eliminating linear trajectory redundancy, enabling cross-trajectory reuse, and supporting shortest / most reliable path retrieval based on edge weights. 2. Multimodal state normalization and semantic edge representation: By fusing similarity calculations, cross-trajectory page screenshot merging and deduplication are achieved, triggering manual review in low-confidence or conflict situations. Simultaneously, each connecting edge is semantically described (operation type, operation element, operation position, input parameters, etc.) and assigned statistical weights (frequency, success rate, average time consumption, etc.), providing a reliable basis for training sample weighting, path optimization, and anomaly detection, improving model generalization and interpretability. 3. Graph-driven trajectory synthesis-replay-closed-loop lifecycle management: Based on heuristic path search on the graph, executable trajectories are automatically synthesized and compensation actions are inserted where necessary to ensure runnability. The trajectory is replayed and verified on real devices or simulators, and the results are fed back to the relationship graph to update edge weights, add or remove nodes / edges, and generate migration suggestions. This forms a closed-loop governance process of generation-verification-correction, which significantly reduces manual maintenance costs and continuously improves the execution reliability and data quality of GUI agents.

[0100] Corresponding to the above-described method for managing user operation data, one embodiment of this specification also provides an apparatus for managing user operation data, such as... Figure 9 As shown, the device may include:

[0101] The acquisition unit 902 is used to acquire multiple operation trajectories, wherein any first operation trajectory is obtained based on the target intelligent agent's operation of the first application for the first task. The first operation trajectory includes page information of multiple pages arranged in sequence and the operation to be performed for each jump between two adjacent pages.

[0102] The construction unit 904 is used to construct a relationship graph based on multiple operation trajectories. The relationship graph includes multiple nodes and connecting edges between nodes. Each node corresponds to one of the multiple pages operated by the multiple operation trajectories, and the connecting edge corresponds to the operation between two adjacent pages included in the multiple operation trajectories.

[0103] In one embodiment, the above page information is a page screenshot; the construction unit 904 includes:

[0104] The merging submodule 9042 is used to select similar page screenshots from any two operation trajectories and merge them to obtain a merged screenshot by calculating the similarity between the page screenshots included in the two operation trajectories.

[0105] Update submodule 9044, which is used to update the two operation trajectories mentioned above using the merged screenshot, to obtain two updated trajectories;

[0106] Submodule 9046 is constructed to build a relationship graph based on each updated trajectory determined for multiple operation trajectories.

[0107] In one embodiment, the two operation trajectories mentioned above include a first operation trajectory and a second operation trajectory;

[0108] Merge submodule 9042 is specifically used for:

[0109] For any screenshot of the first page in the first operation trajectory, based on the image representation of the first page screenshot and the image representation of each page screenshot in the second operation trajectory, calculate the similarity between the first page screenshot and each page screenshot; or,

[0110] Based on the text representation of the page path in the first page screenshot and the text representation of the page path in each of the other page screenshots, the similarity between the first page screenshot and each of the other page screenshots is calculated.

[0111] In one embodiment, the page information includes a page screenshot and a page path, and the device further includes:

[0112] The embedding processing unit 906 is used to embed a screenshot of the page corresponding to the node into a single node using an image encoder to obtain an image representation of the node; and to embed the page path of the page corresponding to the node into a text representation of the node using a text encoder.

[0113] Storage unit 908 is used to store multiple nodes and the connection edges between nodes, as well as the associated storage of the corresponding image representation and text representation for a single node.

[0114] In one embodiment, storage unit 908 is further configured to associate storage with corresponding node attributes for a single node, the node attributes including one or more of the following: page path, application identifier, environment context, and metadata, etc.

[0115] In one embodiment, the storage unit 908 is further configured to associate and store the corresponding edge attributes for a single connection edge, the edge attributes including one or more of the following: operation element, execution time, operation status, operation position, input parameters, and termination node, etc.

[0116] In one embodiment, storage unit 908 is specifically used for:

[0117] Calculate and store the edge weights of the connecting edges, where the edge weights are calculated based on one or more of the following: the execution frequency of the corresponding operation, the success rate of the subsequent path, and the execution time of the corresponding operation.

[0118] In one embodiment, the device further includes:

[0119] Synthesis unit 910 is used to synthesize target trajectories based on relational graphs;

[0120] The verification unit 912 is used to replay and verify each target operation included in the target trajectory using real equipment or a simulator; the replay verification for a single target operation includes determining whether the execution result of the target operation is correct.

[0121] Training unit 914 is used to train the target agent based on the target trajectory after the playback verification is passed.

[0122] In one embodiment, the synthesis unit 910 is specifically used for:

[0123] The target node corresponding to the merged screenshot is determined from the relationship graph. Based on the first preceding path and the second following path of the target node, the target trajectory is synthesized. The pages covered by the first preceding path and the pages covered by the second following path belong to different operation trajectories.

[0124] In one embodiment, the device further includes:

[0125] The input unit 916 is used to input the target prompt word into the target agent. The target prompt word is used to indicate that the first and last nodes are determined from the relationship graph for the user question, and the target path is determined based on the edge weight or execution time of each connection edge between the first and last nodes. The answer to the user question is output based on the target path.

[0126] In one embodiment, the device further includes:

[0127] Update unit 918 is used to update the target node and / or target connection edge in the relationship graph when the application interface or function changes.

[0128] The verification unit 912 is also used to replay and verify the first operation related to the target node and / or the second operation corresponding to the target connection edge using real equipment or a simulator, including determining whether the execution result of the first and / or second operation is correct.

[0129] In one embodiment, the above operation belongs to one of the following operation categories: click operation, swipe operation, long press operation, input operation, and selection operation, etc.

[0130] This specification provides an embodiment of a device for managing user operation data, which can construct a unified relationship graph based on multiple operation trajectories and perform centralized management on this graph, thereby significantly improving the utilization efficiency of operation data and greatly enhancing the maintainability of operation data.

[0131] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 3 The method described.

[0132] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 3 The method described.

[0133] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the medium or device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0134] The steps of the methods or algorithms described in conjunction with the disclosure in this specification can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, external hard disk, CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a server. Of course, the processor and storage medium can also exist as discrete components in the server.

[0135] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0136] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26k20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0137] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0138] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0139] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0140] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0141] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0143] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0144] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0145] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0146] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0148] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0149] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for managing user operation data, comprising: Obtain multiple operation trajectories; The first operation trajectory is obtained based on the target agent's operation on the first application for the first task. The first operation trajectory includes page information of multiple pages arranged in sequence and the operation to be performed for each jump between two adjacent pages. A relationship graph is constructed based on the multiple operation trajectories. The relationship graph includes multiple nodes and connecting edges between nodes. Each node corresponds to one of the multiple pages operated by the multiple operation trajectories, and the connecting edge corresponds to the operation between two adjacent pages included in the multiple operation trajectories.

2. The method according to claim 1, wherein, The page information is a screenshot; the construction of a relationship graph based on the multiple operation trajectories includes: For any two operation trajectories, the similarity between the page screenshots included in the two operation trajectories is calculated, and similar page screenshots are selected from the two operation trajectories and merged to obtain a merged screenshot; The two operation trajectories are updated using the merged screenshot to obtain two updated trajectories; A relationship graph is constructed based on each updated trajectory determined for the multiple operation trajectories.

3. The method according to claim 2, wherein, The two operation trajectories include a first operation trajectory and a second operation trajectory; The calculation of the similarity between the page screenshots included in the two operation trajectories includes: For any first page screenshot in the first operation trajectory, the similarity between the first page screenshot and each page screenshot in the second operation trajectory is calculated based on the image representation of the first page screenshot and the image representation of each page screenshot in the second operation trajectory. or, Based on the text representation of the page path of the first page screenshot and the text representation of the page path of each of the page screenshots, the similarity between the first page screenshot and each of the page screenshots is calculated.

4. The method according to claim 1, wherein, The page information includes a page screenshot and a page path; the method further includes: For a single node, an image encoder is used to embed a screenshot of the page corresponding to the node to obtain the image representation of the node; a text encoder is used to embed the page path of the page corresponding to the node to obtain the text representation of the node. The system stores the multiple nodes and the connection edges between them, as well as the corresponding image representation and text representation for each individual node.

5. The method according to claim 4, further comprising: For a single node, the corresponding node attributes are associated with it; the node attributes include one or more of the following: page path, application identifier, environment context, and metadata.

6. The method according to claim 4, further comprising: For a single connected edge, the corresponding edge attributes are associated and stored; the edge attributes include one or more of the following: operation element, execution time, operation status, operation position, input parameters, and termination node.

7. The method according to claim 4, wherein, The storage of the relationship graph also includes: Calculate and store the edge weights of the connected edges; wherein the edge weights of the connected edges are calculated based on one or more of the following: the execution frequency of the corresponding operation, the execution success rate of the subsequent path, and the execution time of the corresponding operation.

8. The method according to claim 2, further comprising: Based on the aforementioned relationship graph, the target trajectory is synthesized; Using real equipment or a simulator, the target operations included in the target trajectory are replayed and verified. Replay verification for a single target operation includes determining whether the execution result of the target operation is correct; After the playback verification is passed, the target agent is trained based on the target trajectory.

9. The method according to claim 8, wherein, The synthesized target trajectory includes: The target node corresponding to the merged screenshot is determined from the relationship graph. Based on the first preceding path and the second following path of the target node, the target trajectory is synthesized. The page covered by the first preceding path and the page covered by the second following path belong to different operation trajectories.

10. The method of claim 8, further comprising: Input the target prompts into the trained target agent; The target prompt is used to indicate that the first and last nodes are determined from the relationship graph in response to the user's question, and the target path is determined based on the edge weight or execution time of each connecting edge between the first and last nodes. The answer to the user's question is then output based on the target path.

11. The method according to claim 1, further comprising: When the application interface or function changes, update the target nodes and / or target connection edges in the relationship graph; Using real equipment or a simulator, replay and verify the first operation related to the target node and / or the second operation corresponding to the target connection edge, including determining whether the execution results of the first and / or second operations are correct.

12. The method according to claim 1, wherein, The operation belongs to one of the following operation categories: click operation, swipe operation, long press operation, input operation, and selection operation.

13. An apparatus for managing user operation data, comprising: The acquisition unit is used to acquire multiple operation trajectories; The first operation trajectory is obtained based on the target agent's operation on the first application for the first task. The first operation trajectory includes page information of multiple pages arranged in sequence and the operation to be performed for each jump between two adjacent pages. A construction unit is configured to construct a relationship graph based on the multiple operation trajectories. The relationship graph includes multiple nodes and connecting edges between nodes. Each node corresponds to one of the multiple pages operated by the multiple operation trajectories, and the connecting edge corresponds to the operation between two adjacent pages included in the multiple operation trajectories.

14. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-12.

Images