GUI (Graphical User Interface) interaction data generation method, training system and electronic equipment

Abstract

The invention provides a GUI (Graphical User Interface) interaction data generation method. The method comprises the following steps: acquiring a demand document; the demand document is used for describing a service demand of a training scene corresponding to a first service program constructed based on LLM; the first service program is used for operating the GUI based on an operation instruction input by a user; extracting structured service semantics related to the service demand from the demand document, and automatically developing a GUI (Graphical User Interface) based on the service semantics; an operable GUI component contained in the GUI is bound with service semantics; an acquisition code is inserted into an execution code corresponding to the GUI component; in response to an operation on the GUI component contained in the deployed GUI, executing the collection code inserted in the execution code corresponding to the GUI component, so as to collect GUI interaction data generated by operating the GUI component according to service semantics; the GUI interaction data is used as a training sample to train the first service program.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a GUI interactive data generation method, training system and electronic device. Background Technology

[0002] With the improvement of LLM (Large Language Model) capabilities, training an LLM that can autonomously operate a GUI (Graphical User Interface) has gradually become a reality.

[0003] However, effective training of this type of LLM not only requires a massive amount of GUI interaction data as training samples, but also requires that this GUI interaction data not only record the user's operation sequence on the interface, but also contain the service semantics behind these operations.

[0004] It is evident that training this type of LLM typically requires high-quality training samples, which means that the inability to obtain a large number of high-quality training samples has become the main limitation in training this type of LLM. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a method for generating GUI interactive data is provided, the method comprising: Obtain the requirements document; wherein the requirements document describes the service requirements of the training scenario corresponding to the first service program built based on LLM; the first service program is used to operate the GUI based on user input operation instructions; Structured service semantics related to the service requirements are extracted from the requirements document, and a GUI is automatically developed based on the service semantics; wherein, the operable GUI components contained in the GUI are bound to the service semantics, and data collection code is inserted into the execution code corresponding to the GUI components; the data collection code is used to collect GUI interaction data generated by operating the GUI components in accordance with the service semantics. In response to an operation on the GUI components included in the deployed GUI, the collection code inserted in the execution code corresponding to the GUI component is executed to collect GUI interaction data generated by the operation on the GUI component according to the service semantics bound to the GUI component; wherein, the GUI interaction data is used as training samples to train the first service program.

[0006] Optionally, extract structured service semantics related to the service requirements from the requirements document, and automatically develop a GUI based on the service semantics, including: The requirement document is input into a second service program built on an LLM basis, which extracts structured service semantics related to the service requirements from the requirement document and automatically develops a GUI based on the service semantics; wherein, the second service program is used for automated GUI development.

[0007] Optionally, the service semantics include structured service metadata consisting of service entities extracted from the requirements document, attributes of the service entities, and relationships between the service entities; and an operation list defined for the service entities extracted from the requirements document. Accordingly, the operable GUI components included in the GUI are bound to the service metadata and the operation list.

[0008] Optionally, in response to an operation on a GUI component included in the deployed GUI, the data collection code inserted into the execution code corresponding to the GUI component is run to collect GUI interaction data generated by operating the GUI component according to the service semantics bound to the GUI component, including: The third service program built on LLM performs user operation simulation on the deployed GUI; In response to the operation of the GUI components contained in the GUI simulated by the third service program, the execution code corresponding to the GUI component is executed, and during the execution of the execution code, the collection code inserted in the execution code is further executed, so as to collect the GUI interaction data generated by the execution of the execution code according to the service semantics bound to the GUI component.

[0009] Optionally, the GUI interaction data includes: Operation timestamps for operations performed on the GUI components; The operation type performed on the GUI component; The page state changes generated by the GUI after the operation of the GUI component; Service semantic information corresponding to the service semantics bound to the GUI component; The data state changes that occur after the GUI component is operated on.

[0010] Optionally, when a third service program built on LLM performs user operation simulation on the deployed GUI, the GUI interaction data also includes the reasoning process of the third service program during the user operation simulation on the deployed GUI.

[0011] Optionally, the service program built on LLM includes an agent built on LLM; the acquisition code includes embedded code.

[0012] According to a first aspect of one or more embodiments of this specification, a training system is also provided, comprising: The first service program, built on LLM, operates the GUI based on user-input commands; The second service program, built on an LLM framework, extracts structured service semantics related to the service requirements from a requirements document and automatically develops a GUI based on the service semantics. The requirements document describes the service requirements for a training scenario corresponding to the first service program. The operable GUI components within the GUI are bound to the service semantics, and data collection code is inserted into the execution code corresponding to each GUI component. The data collection code is used to collect GUI interaction data generated by operating the GUI components according to the service semantics. The data acquisition subsystem, in response to an operation on the GUI components contained in the deployed GUI, executes the data acquisition code inserted in the execution code corresponding to the GUI component, so as to collect GUI interaction data generated by the operation on the GUI component according to the service semantics bound to the GUI component; The training subsystem uses the GUI interaction data as training samples to train the first service program.

[0013] According to a third aspect of one or more embodiments of this specification, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the executable instructions to implement the steps of the method as described in any of the first aspects above.

[0014] According to a fourth aspect of one or more embodiments of this specification, a computer program product is also provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the first aspects above.

[0015] In the above embodiments, on the one hand, by extracting structured service semantics from the requirements document and forcibly binding the service semantics to the generated GUI components, it can be ensured that the final collected GUI interaction data naturally contains real service semantics. This allows the collected GUI interaction data to be directly used as training samples to train LLM-based service programs that operate the GUI based on user input commands. This provides automated generation of high-quality training samples for such service programs, achieving end-to-end automated generation of training samples. This not only lowers the training threshold for such service programs but also solves the problems of not being able to obtain enough high-quality training samples containing service semantics and the high cost of manually annotating training samples during the training process.

[0016] On the other hand, since the requirements document can describe the service requirements of the training scenario corresponding to this type of service program, based on the service semantics extracted from the requirements document, a runnable GUI with built-in data acquisition capabilities can be automatically developed. This is equivalent to automatically generating a runnable simulated training environment for this type of service program. Furthermore, the training samples automatically collected in this simulated training environment can more accurately reflect the user's operational logic in the real operation scenario of the GUI, thereby improving the training effect of this type of service program and enhancing the accuracy and generalization ability of this type of service program in completing operations. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the architecture of a GUI interactive data generation system provided in an exemplary embodiment; Figure 2 This is a flowchart illustrating a GUI interactive data generation method provided in an exemplary embodiment; Figure 3 This is an exemplary embodiment of a flowchart illustrating the end-to-end collection of GUI interaction data based on a requirements document; Figure 4 This is an exemplary embodiment of an end-to-end training system; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment; Figure 6 This is a block diagram of a GUI interactive data generation apparatus provided in an exemplary embodiment. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0019] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0020] While current mainstream LLM-based generative programming tools all support using natural language editing prompts to guide the LLM in generating code for users and completing GUI development, these generative programming tools primarily serve to assist humans in code development and often suffer from the problem of "emphasizing visual / code implementation while neglecting service semantics".

[0021] When training an LLM that can autonomously operate a GUI (such as an agent built on an LLM), a large amount of high-quality GUI interaction data with service semantics is usually required as training samples.

[0022] However, while current mainstream LLM-based generative programming tools support using natural language editing prompts to guide the LLM in generating code for users and completing GUI development, these generative programming tools primarily serve to assist humans in code development and often suffer from the problem of "emphasizing visual / code implementation while neglecting service semantics".

[0023] This means that while generative programming tools can develop a GUI that meets user needs, they cannot fully utilize the generative capabilities of LLM itself to autonomously generate enough high-quality training samples for the training of such LLMs.

[0024] Based on this, this specification proposes a technical solution that can automatically and end-to-end generate high-quality training samples with service semantics for LLM service programs that operate on a GUI based on user input commands.

[0025] In this technical solution, firstly, a requirements document describing the service requirements corresponding to the training scenario built on LLM can be obtained; wherein, the first service program can be used to operate the GUI based on user input operation instructions; Secondly, structured service semantics related to the aforementioned service requirements can be extracted from the requirements document, and a GUI can be automatically developed based on these service semantics. The operable GUI components contained in the GUI are bound to the aforementioned service semantics, and data collection code is inserted into the execution code corresponding to the GUI components. This data collection code is used to collect GUI interaction data generated by operating the GUI components in accordance with the aforementioned service semantics. Furthermore, once the GUI is deployed, in response to operations on the GUI components contained in the GUI, the collection code inserted in the execution code corresponding to the GUI component can be executed to collect the GUI interaction data generated by the operation on the GUI component according to the service semantics bound to the GUI component.

[0026] Finally, the automatically generated GUI interaction data can be used as training samples to train the first service program mentioned above.

[0027] In the above technical solution, on the one hand, by extracting structured service semantics from the requirements document and forcibly binding the service semantics to the generated GUI components, it can be ensured that the final collected GUI interaction data naturally contains real service semantics. This allows the collected GUI interaction data to be directly used as training samples to train LLM-based service programs that operate the GUI based on user input commands. This provides automated generation of high-quality training samples for such service programs, achieving end-to-end automated generation of training samples. This not only lowers the training threshold for such service programs but also solves the problems of not being able to obtain enough high-quality training samples containing service semantics and the high cost of manual annotation of training samples during the training process.

[0028] On the other hand, since the requirements document can describe the service requirements of the training scenario corresponding to this type of service program, based on the service semantics extracted from the requirements document, a runnable GUI with built-in data acquisition capabilities can be automatically developed. This is equivalent to automatically generating a runnable simulated training environment for this type of service program. Furthermore, the training samples automatically collected in this simulated training environment can more accurately reflect the user's operational logic in the real operation scenario of the GUI, thereby improving the training effect of this type of service program and enhancing the accuracy and generalization ability of this type of service program in completing operations.

[0029] Figure 1 This is a schematic diagram of the architecture of a GUI interactive data generation system provided in an exemplary embodiment.

[0030] like Figure 1 As shown, the system may include a server 11, a network 12, and several electronic devices, such as a PC (Personal Computer) 13, a mobile phone 14, etc.

[0031] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted in a host cluster. During operation, server 11 can run a server-side program to implement functions related to GUI interactive data generation. For example, when server 11 runs the program, it can act as the server-side of the GUI interactive data generation system.

[0032] PC23 and mobile phone 14 are just some of the types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. During operation, the electronic device can also run a client-side program to implement functions related to GUI interactive data generation. For example, when the electronic device runs the program, it can act as a client of the GUI interactive data generation system.

[0033] The client-side application of the aforementioned GUI interactive data generation system can be launched and run on an electronic device. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form. Alternatively, when using web technologies such as HTML5, the relevant functions can be implemented through a browser-displayed page. This browser can be a standalone browser application or a browser module embedded within some applications.

[0034] As for the network 12 that enables interaction between electronic devices such as PC13 and mobile phone 14 and server 11, communication can be achieved using either wired or wireless networks, depending on the communication methods supported by the respective electronic devices. This specification does not impose any restrictions on this. For example, PC13 can support both wired and wireless communication, so it can use either wired or wireless networks as needed. Mobile phone 14 typically only supports wireless communication, so it can use a wireless network for communication.

[0035] The technical solution of this specification will be described in detail below with reference to the accompanying drawings.

[0036] Please see Figure 2 , Figure 2 This is a flowchart illustrating a GUI interactive data generation method, which includes the following execution process: Step 202, obtain the requirements document; wherein, the requirements document is used to describe the service requirements of the training scenario corresponding to the first service program built based on LLM; the first service program is used to operate the GUI based on the operation instructions input by the user; The aforementioned service programs may specifically include artificial intelligence service programs built on LLM.

[0037] In some embodiments, the entity executing the above method may specifically be a service program deployed on a travel service platform.

[0038] The aforementioned service program can specifically include any form of intelligent service program. In practical applications, the service program can also include server-side programs and client-side programs, which work together to provide platform services to users.

[0039] In some embodiments, the aforementioned service program may specifically include an artificial intelligence service program built on an LLM-based architecture.

[0040] Specifically, AI service programs built on LLM can include any form of application that uses LLM as the core computing engine and is responsible for providing natural language understanding, generation, reasoning, and decision-making capabilities.

[0041] In some embodiments, the aforementioned AI service program built on LLM may specifically include an agent built on LLM. The agent can use LLM as a core component to autonomously perform tasks and make decisions in a specific environment.

[0042] For example, the first service program mentioned above can specifically be an intelligent agent that uses LLM as its core component and has the ability to operate the GUI based on user input commands. The GUI mentioned above can be a web-based GUI or a native GUI within a mobile operating system app.

[0043] The aforementioned requirements document may specifically include any form of document that can describe the service requirements of the training scenario corresponding to the aforementioned first service program.

[0044] For example, the aforementioned requirements document can be something like a PRD (Product Requirement Document), which can describe the product's service goals, functional requirements, user scenarios, and constraints, etc.

[0045] It should be noted that, since the aforementioned first service program needs to simulate user interactions with the GUI in all scenarios, it typically requires training on a massive number of training scenarios. In this case, before training the first service program, a requirements document describing the service needs of different training scenarios can be obtained as input data for automatically generating training samples.

[0046] Specifically, the aforementioned training scenario refers to a simulation environment in which the first service program (e.g., an agent) to be trained needs to learn how to perform operations with specific service semantics within a GUI. The service requirements of this training scenario can be defined by a requirements document and automatically constructed into an interactive GUI (e.g., a web application) through subsequent processes.

[0047] In practical applications, the training scenario described above can specifically be a well-defined service domain. This service domain should be able to be abstracted into specific service semantics, and the abstracted service semantics should be instantiated into an interactive interface that can bind to the service semantics GUI components. Any service domain that meets these characteristics can constitute a specific training scenario.

[0048] For example, in one instance, the training scenario described above could be an e-commerce product guide and purchase scenario. In such a scenario, it is necessary to build a GUI for an e-commerce platform that is geared towards consumers, and to simulate user operations such as browsing the product list displayed in the GUI, searching for products by keywords, filtering products using conditions such as price and category, viewing product details, and adding selected products to the shopping cart.

[0049] In another example, the training scenario described above could specifically be a social media content browsing and interaction scenario. In this scenario, it is necessary to build a user-facing GUI for a social platform and simulate user actions such as browsing posts displayed in the GUI interface, liking, saving, commenting on posts, and following other users.

[0050] When obtaining requirement documents to describe the service requirements of different training scenarios, you can directly read existing requirement documents from a specified location, or you can automatically generate requirement documents based on the service requirements of a specific training scenario.

[0051] For example, in some embodiments, users can input service requirements for a specific training scenario as prompts into an LLM or an agent built on an LLM. The LLM or the agent built on an LLM can then perform inference calculations based on the prompts to automatically generate a requirement document that describes the service requirements of the training scenario.

[0052] Step 204: Extract structured service semantics related to the service requirements from the requirements document, and automatically develop a GUI based on the service semantics; wherein, the operable GUI components contained in the GUI are bound to the service semantics, and data collection code is inserted into the execution code corresponding to the GUI components; the data collection code is used to collect GUI interaction data generated by operating the GUI components in accordance with the service semantics.

[0053] Once the requirements document is obtained, structured service semantics related to the service requirements of the training scenario can be extracted from the document, and then an automated programming tool can be used to automatically develop a GUI based on the service semantics.

[0054] In practical applications, since requirement documents are often unstructured, the unstructured parsing capabilities and code development capabilities of LLM can be used to extract structured service semantics related to the service requirements of the training scenario mentioned above from the requirement document, and a GUI can be automatically developed based on the extracted service semantics.

[0055] Please see Figure 3 , Figure 3 This specification illustrates a flowchart for end-to-end collection of GUI interactive data based on a requirements document.

[0056] In some embodiments, such as Figure 3 As shown, during the process of automatically generating GUI interaction data for the first service program as training samples, the obtained requirement documents can be further input into the second service program built on LLM.

[0057] Specifically, the aforementioned second service program can also be an artificial intelligence service program built on an LLM (Liquidity Management System) architecture. For example, this second service program could be an intelligent agent for automating the development of a GUI (User Interface).

[0058] After receiving the input requirement document, the second service program can parse the document and extract structured service semantics related to the service requirements of the training scenario based on the parsing results. For example, service semantics in JSON format.

[0059] The service semantics extracted from the aforementioned requirements document are usually related to the service requirements of the training scenario. Specifically, they may include a set of information that can fully define the data composition and operation logic of the training scenario.

[0060] In some embodiments, the above service semantics may specifically include the following two core components: First, structured service metadata.

[0061] The aforementioned service metadata can be used to describe the static data elements involved in the training scenario. In practical applications, the service metadata can specifically include three parts: service entities, attributes of service entities, and relationships between service entities.

[0062] Secondly, the operation list.

[0063] The above list of operations specifically describes the operations that are allowed to be performed in this training scenario. In practical applications, the operations in the above list can define the core service operations that can be performed on the aforementioned service entities.

[0064] It should be noted that the specific data format of the operations in the above operation list is not specifically limited in this specification; for example, it may be descriptive information related to the operation, or it may be a pre-developed and usable API or function based on these operations.

[0065] Please continue reading Figure 3 Once the second service program extracts the structured service semantics related to the service requirements of the training scenario from the requirements document, it can further automate the development of the GUI based on the service semantics.

[0066] Specifically, the operable GUI components included in the GUI automatically developed based on structured service semantics by the second service program can be bound to the aforementioned service semantics. Furthermore, to achieve automatic collection of GUI interaction data, collection code can be inserted into the execution code corresponding to the operable GUI components within the GUI. This collection code can be used to collect GUI interaction data generated by operating the GUI components according to the aforementioned service semantics.

[0067] In some embodiments, please continue to see Figure 3 The second service procedure described above, based on the structured service semantics extracted from the requirements document, automates the GUI development process and may include the following steps: Step 1: Structural planning for generating a graphical user interface based on service semantics; In this step, firstly, the second service program can determine the set of fields that need to be displayed and entered in the GUI based on the service entities and attributes of each service entity contained in the service metadata. Secondly, based on the business operations defined in the above operation list, the set of interactive triggering functions that need to be provided in the GUI can be determined; Furthermore, based on preset interface organization rules, the above-mentioned field set and function set can be integrated and laid out to plan an interface structure that includes at least one page, and each page contains at least one operable GUI component.

[0068] Finally, in the above planning process, we can also establish the binding relationship between the operable GUI components and the above service semantics and business logic.

[0069] For example, on the one hand, a binding relationship can be established between GUI components used for data display or entry and specific attributes of service entities specified in the aforementioned service metadata; on the other hand, a binding relationship can be established between GUI components used to trigger interactions and specified business operations in the aforementioned operation list.

[0070] Step 2: Generate executable code based on the above structural plan and inject the acquisition code synchronously.

[0071] In this step, firstly, the second service program can generate front-end rendering code corresponding to the GUI interface structure presented in the above structural plan. Secondly, the second service program can also generate back-end service interface code corresponding to each business operation in the above operation list.

[0072] Finally, during the process of generating the front-end presentation code and the back-end service interface code, predefined collection code can be synchronously inserted into the predetermined position of the generated code based on the above binding relationship.

[0073] For example, you can inject collection code into the event response logic code of a GUI component that is bound to a specific business operation to record the business operation being called and the input parameters related to this call; you can also inject collection code into the backend service interface code to record the execution process and data status changes of the interface.

[0074] The specific form of the above-mentioned data collection code is not specifically limited in this specification; for example, in one example, the above-mentioned data collection code may specifically include data tracking code.

[0075] Step 206: In response to an operation on the GUI components included in the deployed GUI, execute the collection code inserted in the execution code corresponding to the GUI component to collect GUI interaction data generated by operating the GUI component according to the service semantics bound to the GUI component; wherein, the GUI interaction data is used as training samples to train the first service program.

[0076] Please continue reading Figure 3 Once the GUI is automatically developed based on the service semantics extracted from the requirements document, it can be used as a training environment for the simulation corresponding to the first service program and deployed to the specified runtime environment; for example, the GUI can be deployed to the training environment for training the first service program.

[0077] Once the GUI is deployed, the operation of the operable GUI components can trigger the execution of the pre-inserted collection code in the execution code corresponding to the GUI component, thereby automatically generating high-quality GUI interaction data containing service semantics as training samples for the first service program.

[0078] It should be noted that the specific methods for operating the operable GUI components contained in the deployed GUI are not specifically limited in this specification.

[0079] In some embodiments, please continue to see Figure 3 A third-party service program based on LLM can be introduced, and user operation simulation can be performed on the deployed GUI through this third-party service program.

[0080] Specifically, the aforementioned third service program can also be an artificial intelligence service program built on an LLM (Liquidity Management System). For example, this third service program can be an intelligent agent used for user operation simulation.

[0081] In this scenario, once the GUI is deployed, the third service program can be run to simulate user operations on the deployed GUI. Then, in response to the simulated operations on the GUI components within the GUI, the execution code corresponding to the GUI component can be executed. During the execution of this code, the aforementioned data collection code (such as event tracking code) inserted within it can be further executed to collect GUI interaction data generated by the execution code according to the service semantics bound to the GUI component.

[0082] Specifically, collecting GUI interaction data generated by executing the code according to the service semantics bound to the GUI component means collecting the GUI interaction data generated by executing the code according to the constraints embodied in the service semantics. This method prevents the collected GUI interaction data from deviating from the service semantics.

[0083] Of course, in practical applications, besides using a service program built on LLM to simulate user operations on the deployed GUI and operate the operable GUI components contained in the GUI, other methods can also be used to operate the operable GUI components contained in the GUI. For example, the operable GUI components contained in the deployed GUI can be operated manually, or the operable GUI components contained in the GUI can be operated by running static operation rules, etc.

[0084] The aforementioned GUI interaction data can specifically be a structured sequence of data, which can completely record every meaningful interaction that occurs in the GUI and the context information related to that interaction.

[0085] In some embodiments, the GUI interaction data described above includes the following components: The first part consists of operation timestamps for manipulating GUI components. This data specifically records the precise moments when operations are performed on the operable GUI components contained within the GUI.

[0086] The second part describes the operation types performed on this GUI component. This data can specifically record the type of operation performed on the operable GUI components contained in the GUI component; for example, it can include operations such as clicking, selecting, etc.

[0087] The third part describes the page state changes that occur after the GUI component is manipulated. This data can specifically record all visual changes that occur in the GUI component before and after an operation is performed on it; for example, the page state change can be a page snapshot (such as a screenshot) after an operation is performed on the GUI component.

[0088] Part Four: Service semantic information corresponding to the service semantics bound to this GUI component; This data specifically records the service semantics triggered by operations performed on the GUI component.

[0089] For example, taking the above service semantics, which include the above service metadata and the above operation list, as an example, this part of the data can specifically record the impact of the operation performed on the GUI component or the business entities involved, as well as the business operations (such as APIs) triggered by the operation.

[0090] Part 5 describes the changes in data state after operations are performed on the GUI component.

[0091] This data specifically records changes in the underlying system data state after operations are performed on the GUI component. For example, in practical applications, this data may include data returned by API calls.

[0092] In some embodiments, if a third service program based on LLM is used to simulate user operation of the deployed GUI, the GUI interaction data may include, in addition to the five parts shown above, the reasoning process of the third service program in simulating user operation of the deployed GUI.

[0093] This data section specifically records the internal reasoning process (Chain-of-Thought) of the aforementioned third service program when making specific operational decisions during user operation simulation. For example, it can record the thinking and reasoning process by which the agent infers the operation to be performed on the GUI component.

[0094] Once the GUI interaction data generated by operating the GUI component is collected, the GUI interaction data contains service semantics. This GUI interaction data can be used as a training sample to further train the first service program mentioned above.

[0095] The training method used to train the first service program is not specifically limited in this specification. In practical applications, it can be flexibly selected based on specific needs.

[0096] For example, in practical applications, the first service program mentioned above can be trained using imitation learning or reinforcement learning methods.

[0097] It should be noted that since the above requirements document can describe the service requirements of the training scenario corresponding to the first service program, based on the service semantics extracted from the requirements document, a runnable GUI with built-in data acquisition capabilities is automatically developed, which is equivalent to automatically generating a runnable simulation training environment for this type of service program.

[0098] In practical applications, GUI sequence data can be automatically collected and generated in the training environment of this simulation to perform end-to-end training of the first service program mentioned above.

[0099] Please see Figure 4 , Figure 4 This specification illustrates an end-to-end training system.

[0100] like Figure 4 As shown, the training system described above may include: The first service program, built on LLM, is used to operate the GUI based on user-input commands. A second service program built on LLM is used to extract structured service semantics related to the service requirements from a requirements document, and to automatically develop a GUI based on the service semantics. The requirements document describes the service requirements of a training scenario corresponding to the first service program. The operable GUI components included in the GUI are bound to the service semantics, and data collection code is inserted into the execution code corresponding to each GUI component. The data collection code is used to collect GUI interaction data generated by operating the GUI components according to the service semantics. A data acquisition subsystem is configured to, in response to an operation on a GUI component contained in the deployed GUI, execute the data acquisition code inserted in the execution code corresponding to the GUI component, so as to acquire GUI interaction data generated by the operation on the GUI component in accordance with the service semantics bound to the GUI component; The training subsystem is used to train the first service program using the GUI interaction data as training samples.

[0101] Through the collaborative efforts of the above four parts, a fully automated closed-loop process can be achieved, from describing business requirements to automatically generating training samples containing service semantics, and then training the first service program based on the automatically generated training samples.

[0102] In the above technical solution, on the one hand, by extracting structured service semantics from the requirements document and forcibly binding the service semantics to the generated GUI components, it can be ensured that the final collected GUI interaction data naturally contains real service semantics. This allows the collected GUI interaction data to be directly used as training samples to train LLM-based service programs that operate the GUI based on user input commands. This provides automated generation of high-quality training samples for such service programs, achieving end-to-end automated generation of training samples. This not only lowers the training threshold for such service programs but also solves the problems of not being able to obtain enough high-quality training samples containing service semantics and the high cost of manual annotation of training samples during the training process.

[0103] On the other hand, since the requirements document can describe the service requirements of the training scenario corresponding to this type of service program, based on the service semantics extracted from the requirements document, a runnable GUI with built-in data acquisition capabilities can be automatically developed. This is equivalent to automatically generating a runnable simulated training environment for this type of service program. Furthermore, the training samples automatically collected in this simulated training environment can more accurately reflect the user's operational logic in the real operation scenario of the GUI, thereby improving the training effect of this type of service program and enhancing the accuracy and generalization ability of this type of service program in completing operations.

[0104] Figure 5 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment.

[0105] like Figure 5 As shown, device 500 mainly consists of a communication interface 502, a user interface 504, a processor 506, and a data storage 508. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 510. The communication interface 502 enables device 500 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 502 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 502 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 502 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 502 may also include multiple physical communication interfaces, such as Wi-Fi interfaces, Bluetooth interfaces, and wide-area wireless interfaces.

[0106] User interface 504 includes receiving user input and providing output to the user. Therefore, user interface 504 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 504 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 504 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 500 may support remote access from other devices via communication interface 502 or another physical interface (not shown). User interface 504 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 504 may also be configured as a display device for rendering or displaying text fragments.

[0107] Processor 506 may contain one or more general-purpose processors and / or special-purpose processors.

[0108] Data storage 508 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 506. Data storage 508 may include removable and non-removable components.

[0109] Processor 506 is capable of executing program instructions 518 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 508 to perform the various functions described herein. Data storage 508 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 500, enable device 500 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 518 by processor 506 may result in processor 506 using data 512.

[0110] For example, program instructions 518 may include an operating system 522 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 500 and one or more applications 520 (e.g., a browser, social application, or game application). Similarly, data 512 may include operating system data 516 and application data 514. Operating system data 516 is primarily accessible to the operating system 522, while application data 514 is primarily accessible to one or more applications 520. Application data 514 may reside in a file system visible or hidden from the user of device 500.

[0111] Application 520 can communicate with operating system 522 through one or more application programming interfaces (APIs). These APIs help application 520 read and / or write application data 514, transmit or receive information via communication interface 502, receive or display information on user interface 504, etc.

[0112] In some terminology, application 520 may be simply referred to as "app". Furthermore, application 520 can be downloaded to device 500 through one or more online app stores or app markets. However, applications can also be installed on device 500 in other ways, such as through a web browser or a physical interface on device 500 (e.g., a USB port).

[0113] Please refer to Figure 6 This specification also proposes a GUI interactive data generation device, which can be applied to, for example... Figure 5 The device shown is used to implement the technical solution of this specification. The device may include: The acquisition module 601 acquires a requirements document; wherein, the requirements document is used to describe the service requirements of the training scenario corresponding to the first service program built based on LLM; the first service program is used to operate the GUI based on user input operation instructions; Development module 602 extracts structured service semantics related to the service requirements from the requirements document and automatically develops a GUI based on the service semantics; wherein, the operable GUI components included in the GUI are bound to the service semantics, and data collection code is inserted into the execution code corresponding to the GUI components; the data collection code is used to collect GUI interaction data generated by operating the GUI components according to the service semantics. The acquisition module 603, in response to an operation on the GUI components included in the deployed GUI, executes the acquisition code inserted in the execution code corresponding to the GUI component, so as to acquire GUI interaction data generated by the operation on the GUI component according to the service semantics bound to the GUI component; wherein, the GUI interaction data is used as training samples to train the first service program.

[0114] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0115] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0116] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0117] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0118] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0119] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0120] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0121] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0122] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0123] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0124] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A method for generating GUI interactive data, the method comprising: Obtain the requirements document; wherein the requirements document describes the service requirements of the training scenario corresponding to the first service program built based on LLM; the first service program is used to operate the GUI based on user input operation instructions; Structured service semantics related to the service requirements are extracted from the requirements document, and a GUI is automatically developed based on the service semantics; wherein, the operable GUI components contained in the GUI are bound to the service semantics, and data collection code is inserted into the execution code corresponding to the GUI components; the data collection code is used to collect GUI interaction data generated by operating the GUI components in accordance with the service semantics. In response to an operation on the GUI components included in the deployed GUI, the collection code inserted in the execution code corresponding to the GUI component is executed to collect GUI interaction data generated by the operation on the GUI component according to the service semantics bound to the GUI component; wherein, the GUI interaction data is used as training samples to train the first service program.

2. The method as described in claim 1, comprising extracting structured service semantics related to the service requirements from the requirements document, and automatically developing a GUI based on the service semantics, including: The requirement document is input into a second service program built on an LLM basis, which extracts structured service semantics related to the service requirements from the requirement document and automatically develops a GUI based on the service semantics; wherein, the second service program is used for automated GUI development.

3. The method of claim 2, wherein the service semantics includes structured service metadata consisting of service entities extracted from the requirement document, attributes of the service entities, and relationships between the service entities; and an operation list defined for the service entities extracted from the requirement document. Accordingly, the operable GUI components included in the GUI are bound to the service metadata and the operation list.

4. The method of claim 2, wherein in response to an operation on a GUI component included in the deployed GUI, the collection code inserted into the execution code corresponding to the GUI component is executed to collect GUI interaction data generated by operating the GUI component according to the service semantics bound to the GUI component, including: The third service program built on LLM performs user operation simulation on the deployed GUI; In response to the operation of the GUI components contained in the GUI simulated by the third service program, the execution code corresponding to the GUI component is executed, and during the execution of the execution code, the collection code inserted in the execution code is further executed, so as to collect the GUI interaction data generated by the execution of the execution code according to the service semantics bound to the GUI component.

5. The method as described in claim 4, wherein the GUI interaction data includes: Operation timestamps for operations performed on the GUI components; The operation type performed on the GUI component; The page state changes generated by the GUI after the operation of the GUI component; Service semantic information corresponding to the service semantics bound to the GUI component; The data state changes that occur after the GUI component is operated on.

6. The method as described in claim 5, wherein when a third service program built on an LLM performs user operation simulation on the deployed GUI, the GUI interaction data further includes the reasoning process of the third service program during the user operation simulation on the deployed GUI.

7. The method as described in claim 4, wherein the service program built on LLM includes an agent built on LLM; and the acquisition code includes embedded code.

8. A training system, comprising: The first service program, built on LLM, operates the GUI based on user-input commands; The second service program, built on an LLM framework, extracts structured service semantics related to service requirements from a requirements document and automatically develops a GUI based on these service semantics. The requirements document describes the service requirements for a training scenario corresponding to the first service program. The GUI includes operable GUI components bound to the service semantics, and data collection code is inserted into the execution code corresponding to each GUI component. This data collection code is used to collect GUI interaction data generated by operating the GUI components according to the service semantics. The data acquisition subsystem, in response to an operation on the GUI components contained in the deployed GUI, executes the data acquisition code inserted in the execution code corresponding to the GUI component, so as to collect GUI interaction data generated by the operation on the GUI component according to the service semantics bound to the GUI component; The training subsystem uses the GUI interaction data as training samples to train the first service program.

9. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-7 by executing the executable instructions.

10. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-7.

Images