Interactive instruction refining and executing method and system based on multiple agents

Abstract

The invention discloses an interactive instruction refinement and execution method based on multiple agents. The interactive instruction refinement and execution method is realized through a multi-agent cooperation framework. The framework comprises an execution agent, a verification agent, an information identification agent and a guide agent. The execution agent executes mobile phone operation according to current requirements. The verification agent verifies the accuracy of the operation; the information distinguishing agent extracts information related to the user decision from the application program interface; and the guiding agent actively interacts with the user by using the information to guide the user to clarify the intention of the user. And then entering an interactive refining process, establishing a two-way communication channel between the user and the application program by the system through execution-verification-extraction-guidance circulation, and gradually converting uncertain requirements of the user into clear and executable instructions until the task is completed. The method provided by the invention can effectively deal with complex and real mobile phone use scenes, significantly reduces the difficulty of automatically executing instructions with uncertain requirements, and improves the capability and practicability of intelligent agents.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction and intelligent agent technology, and more specifically, to a multi-agent-based interactive instruction refinement and execution method and system. Background Art

[0002] With the widespread adoption of smartphones, automating natural language commands has become a key research area and a key technology for improving user experience. Users can use voice or text commands to have intelligent agents perform a series of on-screen operations, such as sending messages, booking flights, and shopping online.

[0003] In recent years, large language models have demonstrated powerful capabilities in natural language understanding and reasoning, driving the development of LLM-based smartphone agents. Existing research focuses on processing two types of instructions: The first type is Complete Instructions with Certain Needs (CICN): the user clearly and completely expresses all the necessary information in the instruction, and the agent only needs to translate it into an action sequence step by step, such as "Follow Lei Jun on the Weibo app."

[0004] The second type is incomplete instructions with certain needs (IICN): the user's final need is certain, but some necessary information is missing in the initial instruction. The agent can obtain the missing information through simple follow-up questions. For example, if the user says "send a message to my mother", the agent will ask "what do you want to send?"

[0005] However, existing research has largely overlooked a more common and challenging real-life scenario: incomplete instructions with uncertain needs (IIUN). In this scenario, users don't actively choose not to clearly express their needs; rather, they lack the necessary information to fully define their needs at the initial stage. For example, a user might want to "book a flight from Shanghai to Beijing this Monday." They only know the departure, destination, and approximate date, but not the available flights (such as specific departure times, airlines, and prices). Only after the agent queries and displays the candidate flight information provided by the app can the user further clarify and refine their needs based on this information (for example, choosing "Shenzhen Airlines flight departing at 6:20 PM").

[0006] Existing smartphone agents (whether developed using traditional APIs or LLMs) typically only have a one-way information flow from user to application (user command -> agent analysis -> application action). They lack mechanisms for obtaining feedback from applications and guiding users to refine their needs. They struggle to handle IIUN scenarios because they can't proactively extract the key information needed for user decision-making from the application interface, nor can they effectively interact with users based on this information to clarify their true intent. This limits the application capabilities and user experience of intelligent agents in complex, real-world scenarios.

[0007] Therefore, there is an urgent need for a new method and system that can handle such incomplete instructions with uncertain requirements, establish a two-way communication channel between users and applications, enable intelligent agents to understand application feedback, guide users to refine their requirements, and ultimately complete tasks accurately. Summary of the Invention

[0008] The present invention aims to overcome the shortcomings of existing technologies by providing a method and system for smartphone command execution capable of processing incomplete instructions with uncertain requirements (IIUNs). This method and system aims to transform uncertain requirements into certain requirements through multi-agent collaboration and two-way information interaction with applications, thereby completing user command tasks more realistically and efficiently.

[0009] To achieve the above objectives, the present invention provides a first aspect of a multi-agent-based interactive instruction refinement and execution method, comprising: Receive natural language instructions input by the user and perform preliminary analysis on the received instructions to obtain initial user needs; Based on the current understanding of user needs, perform an action on the target application and generate a natural language description of the action; Based on the generated natural language description and the screen state sequence before and after the operation, evaluate whether the operation is successfully executed and output a result containing verification information or error reasons; In the case where the evaluation operation is successfully executed, content related to the requirement is extracted from the target application according to the current user requirement and the screen status after verification; Based on historical interaction information, current user needs, and extracted content related to the needs, it is determined whether user interaction is required to refine the needs. If user interaction is required to refine the needs, the system receives the refined need information given by the user based on the guided interaction content, and then generates the next operation suggestion based on the updated user needs and historical operations.

[0010] In one embodiment, after receiving a natural language instruction input by a user and performing a preliminary analysis on the received instruction to obtain an initial user requirement, the method further includes: The initial user demand obtained through analysis is stored in a memory module, wherein the memory module stores historical action information, current screen status, optional operation suggestions or error causes.

[0011] In one embodiment, based on the current understanding of user needs, performing an operation on the target application and generating a natural language description of the operation include: Based on the current user's needs, historical action information, current screen status, optional operation suggestions or error causes provided by the memory module, the operation decision function is called to determine and execute the operation on the application interface, and a natural language description of the operation is generated.

[0012] In one embodiment, based on the generated natural language description and the screen state sequence before and after the operation, the operation is evaluated to determine whether it is successfully executed, and a result including verification information or error reasons is output, including: Receive the generated operation description and the screen state sequence before and after the operation, and call the verification function ,Evaluate whether the natural language description is successfully executed.

[0013] In one embodiment, when the evaluation operation is successfully performed, content related to the requirement is extracted from the target application based on the current user requirement and the screen state after verification, including: In the case that the evaluation operation is successfully executed, the information extraction function is called according to the current user demand and the screen status after verification to extract the content related to the demand from the target application and store it in the memory module.

[0014] In one embodiment, when user interaction is required to refine a requirement, receiving the refined requirement information provided by the user based on the guided interaction content, and then generating the next operation suggestion based on the updated user requirement and historical operations, including: When user interaction is required to refine requirements, generate guidance interaction content and display it to the user, so that the user can give detailed requirements based on the guidance content; Call the demand update and suggestion generation function, integrate the refined demand with the user's original demand to obtain the updated user demand, and generate the next operation based on this, and update the updated user demand to the memory module.

[0015] In one embodiment, when user interaction is required to refine a requirement, interactive guidance content is generated and presented to the user, specifically:

[0016] in, Indicates the user's original needs. Represents the sequence of screen states before and after an operation. Indicates the screen state number traversed in this round of operation. Represents historical actions, Indicates the extracted content related to the requirements. Indicates that the output will be provided to the user. Indicates the generated guided interactive content. Indicates the current operation step; Call the demand update and suggestion generation function to integrate the refined demand with the current user's demand to obtain the updated user demand, specifically:

[0017] in, Indicates that the output will be provided to the execution module, Indicates that users provide detailed requirements based on the guidance content. To meet the updated user needs, Provide suggestions for next steps.

[0018] Based on the same inventive concept, the second aspect of the present invention provides a multi-agent-based interactive instruction refinement and execution system, comprising: The instruction receiving and parsing module is used to receive natural language instructions input by the user and perform preliminary parsing on the received instructions to obtain the initial user requirements; The execution module is used to perform operations on the target application based on the current understanding of user needs and generate a natural language description of the operation; The verification module is used to evaluate whether the operation is successfully executed based on the generated natural language description and the screen state sequence before and after the operation, and output a result containing verification information or error reasons; an information identification module, for extracting demand-related content from the target application based on the current user demand and the screen state after verification, if the evaluation operation is successfully executed; The user guidance module is used to integrate historical interaction information, current user needs and extracted content related to the needs, determine whether user interaction is required to refine the needs, and when user interaction is required to refine the needs, receive the refined demand information given by the user based on the guided interaction content, and then generate the next operation suggestion based on the updated user needs and historical operations.

[0019] Based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the multi-agent-based interactive instruction refinement and execution method described in the first aspect.

[0020] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the multi-agent-based interactive instruction refinement and execution method described in the first aspect is implemented.

[0021] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows: The present invention provides a multi-agent-based interactive instruction refinement and execution method, which is implemented through a multi-agent collaborative framework. The framework includes an execution agent, a verification agent, an information identification agent, and a guidance agent: the execution agent performs mobile phone operations according to current needs; the verification agent verifies the accuracy of the operation; the information identification agent extracts information related to the user's decision from the application interface; and the guidance agent uses this information to actively interact with the user and guide the user to clarify his intentions. Then, the interactive refinement process is entered. Through the cycle of execution-verification-extraction-guidance, the system establishes a two-way communication channel between the user and the application, gradually converting the user's uncertain needs into clear and executable instructions until the task is completed. The interactive demand refinement method proposed in the present invention can effectively cope with complex and real mobile phone usage scenarios, significantly reduces the difficulty of automatically executing instructions with uncertain needs, and improves the ability and practicality of intelligent agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 Flowchart of the method for interactive instruction refinement and execution based on multi-agent disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall workflow in an embodiment of the present invention, showing the interaction between the agent modules, the memory module, and the user and the application; Figure 3 Two screen status representation modes supported in the embodiments of the present invention are shown, namely XML mode and SoM mode; Figure 4 This is an example of the workflow of the execution module in a certain interaction step in an embodiment of the present invention; Figure 5 This is an example of the workflow of the verification module in a certain interaction step in an embodiment of the present invention, including both correct and incorrect situations.

[0024] Figure 6 This is an example of the workflow of the information identification module in a certain interaction step in an embodiment of the present invention.

[0025] Figure 7 This is an example of the workflow of the user guidance module in a certain interaction step t in an embodiment of the present invention, showing the output process for the user and the execution module. DETAILED DESCRIPTION

[0026] Example 1 The present invention provides a method and system for interactive demand refinement to execute smartphone commands, aiming to automatically process user commands issued on smartphones, particularly those containing uncertain requirements. This method establishes a closed loop of information interaction between users, applications, and agents through multi-agent collaboration and a memory mechanism. Computer software can be used to support process execution during implementation. Figure 1 The process of the embodiment includes the following steps: S1: Receive natural language instructions input by the user, and perform preliminary analysis on the received instructions to obtain initial user needs.

[0027] Specifically, S1 is to receive and preliminarily parse instructions. The natural language instructions input by the user may contain uncertain requirements. When there are no uncertain requirements, the agent will take over the task and automatically execute the instructions (that is, only repeat steps S2-S3 until the instructions are completed). If there are uncertain requirements, it will enter the subsequent steps (that is, repeat steps S2-S5 until the instructions are completed).

[0028] After receiving the natural language instruction input by the user and performing preliminary analysis on the received instruction to obtain the initial user demand, the method further includes: The initial user demand obtained through analysis is stored in a memory module, wherein the memory module stores historical action information, current screen status, optional operation suggestions or error causes.

[0029] See Figure 2 and Figure 3 ,in, Figure 2 This is a schematic diagram of the overall workflow in an embodiment of the present invention, showing the interaction between the agent modules, the memory module, and the user and the application; Figure 3 Two screen status representation modes supported in the embodiments of the present invention are shown, namely XML mode and SoM mode; S2: Based on the current understanding of user needs, perform an action on the target application and generate a natural language description of the action.

[0030] Specifically, step S2 is to execute an operation, and the operation executed on the target application includes one or more operation steps.

[0031] S2 can be implemented in the following ways: Based on the current user's needs, historical action information, current screen status, optional operation suggestions or error causes provided by the memory module, the operation decision function is called to determine and execute the operation on the application interface, and a natural language description of the operation is generated.

[0032] In the specific implementation process, the process in S2 is formulated as follows:

[0033] in It is the operation The natural language description of Belongs to a predefined operating space , To operate a decision function or execute a proxy, is the current screen status, For the original needs of current users, For optional operation suggestions, For the error reason.

[0034] S3: Based on the generated natural language description and the screen state sequence before and after the operation, evaluate whether the operation is successfully executed and output a result containing verification information or error reasons.

[0035] Specifically, step S3 verifies the operation results. A validation agent (VA) is used to check whether the execution agent's operation was completed as expected and whether the screen status changes as expected. If the operation is incorrect, the cause of the error is recorded and fed back to the execution agent for subsequent decision-making. If the operation is correct, meaning the execution agent's operation is completed and the screen status changes as expected, the process proceeds to the next step.

[0036] S3 can be implemented in the following ways: Receive the generated operation description and the screen state sequence before and after the operation, and call the verification function ,Evaluate whether the natural language description is successfully executed.

[0037] The process in step S3 during the specific implementation process is formulated as follows:

[0038] If the operation is successful, Contains verified operational information (such as ), error cause is empty, Will be used to update the memory module; if the operation fails, is empty, Contains a text message describing the cause of the error, S4: In case the evaluation operation is successfully executed, content related to the requirement is extracted from the target application according to the current user requirement and the screen status after verification.

[0039] Specifically, step S4 is to discern and extract information. When the operation is performed correctly, the discriminant agent (DA) is used to analyze the changes in the application interface (screen state) and extract information content related to the user's potential needs or current task goals.

[0040] S4 can be implemented in the following ways: In the case that the evaluation operation is successfully executed, the information extraction function is called according to the current user demand and the screen status after verification to extract the content related to the demand from the target application and store it in the memory module.

[0041] The process in step S4 during the specific implementation process is formulated as follows:

[0042] in, To extract content related to the needs. For example, on the flight search results page, A list of detailed information may be included for multiple flights. If there is no relevant information on the current screen, = Null.

[0043] S5: Integrate historical interaction information, current user needs, and extracted content related to the needs to determine whether user interaction is required to refine the needs. If user interaction is required to refine the needs, receive the refined demand information given by the user based on the guided interaction content, and then generate the next operation suggestion based on the updated user needs and historical operations.

[0044] Specifically, step S5 guides the user to refine their needs and action suggestions. Guiding interaction content (such as questions or option recommendations) is generated and presented to the user to help the user clarify or refine their uncertain needs. Guiding interaction content can be questions or option recommendations.

[0045] S5 can be achieved in the following ways: When user interaction is required to refine requirements, generate guidance interaction content and display it to the user, so that the user can give detailed requirements based on the guidance content; Call the demand update and suggestion generation function, integrate the refined demand with the user's original demand to obtain the updated user demand, and generate the next operation based on this, and update the updated user demand to the memory module.

[0046] The process in step S5 during the specific implementation is formulated as follows:

[0047]

[0048] in, Indicates that the output will be provided to the user. Indicates that the output will be provided to the execution module.

[0049] The method of the present invention is described below by means of specific examples: Step 1: Receive and initially parse instructions. When the system starts, it first receives the initial instructions input by the user through natural language (voice or text) This instruction usually describes the user's initial intention, but may contain uncertainty or lack all the details required to complete the task (i.e., IIUN scenario). For example, the user may enter "I want to book a flight from Shanghai to Beijing this Monday." The system performs a preliminary analysis of this initial instruction and treats it as the current user demand. (In the first step ) into the memory module. Figure 2 , the user enters the initial requirements ; Step 2: Execute actions based on the current user needs in the memory module , Historical Action Description ,in is the step number, The total number of current steps and the previous operation suggestion provided by the user guidance module and the error reason reported by the verification module , the execution module analyzes the first application screen state of the current step , decide and execute the most appropriate action The function of the execution module can be expressed as:

[0050] in It is the operation The natural language description of Belongs to a predefined operating space , the space contains the following seven actions: Tap(x, y): Perform a click at the screen coordinates (x, y). Swipe(x1, y1, x2, y2): Swipe from (x1, y1) to (x2, y2). Type(text): Enter text text in the currently focused input box. Back: Simulate pressing the system's "Back" key. Home: Return to the device's home screen. GetInfo: A compound action that may include multiple swipes (such as scrolling down a list) to obtain more information on the current screen. It is particularly suitable for browsing lists (such as flights, products). GetBack(x): After executing GetInfo to obtain information and the user makes a selection, return to the screen state of step x (that is, the step before executing GetInfo) to continue subsequent operations. Step 3: Verify the operation results. After the execution module completes the operation, the verification module receives the operation description And a series of screen status representations before and after the operation is performed ,in is the screen state number traversed in this round of operation, The total number of screen states traversed (usually including the state before the operation and the final stable state after the operation). The verification module evaluates whether the operation is performed as described. Successfully executed, and judge whether the screen status changes are in line with expectations. The function of the verification module can be expressed as:

[0051] in To verify the information, To authenticate the agent, if the operation is successful, then Contains verified operational information (such as ), error cause is empty, Will be used to update the memory module; if the operation fails, is empty, Contains text information describing the cause of the error, which will be fed back to the execution module in the next step to assist in error correction.

[0052] Step 4: Identify and extract information. Only when the verification module in step 3 confirms that the operation is successful, the information identification module is started. This module receives the current user needs And the screen state sequence corresponding to this successful operation confirmed by the verification module (Included in The information recognition module analyzes the screen content and extracts the information related to user needs. Related information The process can be expressed as:

[0053] in, For example, on the flight search results page, A list of detailed information may be included for multiple flights. If there is no relevant information on the current screen, = Null. To improve efficiency, this module performs content filtering when processing screen representations (such as XML or SoM). It will be stored in the memory module for use in subsequent steps.

[0054] Step 5: Guide users to refine their needs. The first function of the user guidance module is to determine whether it is necessary and how to guide users. It comprehensively considers the current needs in the memory module. , Historical Actions , Historical Screen Status , and all the content currently extracted If it is determined that the user needs to further clarify the intention, the module will generate guiding interactive content (such as questions, recommendations, requests for clarification, etc.) are presented to the user. This process can be conceptualized as:

[0055] in Indicates that the output will be provided to the user.

[0056] The user receives the guidance information After that, provide feedback, that is, detailed demand information The feedback process can be expressed as:

[0057] The second function of the user guidance module is to receive user feedback and compare it with the previous requirement Merge and update to more specific user needs The module then takes action based on the updated requirements and related historical , generate the next step operation suggestion for the execution module . This process can be expressed as:

[0058] in, Indicates that the output will be provided to the execution module. Updated requirements and operational suggestions is written into the memory module.

[0059] Example 2 Based on the same inventive concept, this embodiment discloses a multi-agent-based interactive instruction refinement and execution system, comprising: The instruction receiving and parsing module is used to receive natural language instructions input by the user and perform preliminary parsing on the received instructions to obtain the initial user requirements; The execution module is used to perform operations on the target application based on the current understanding of user needs and generate a natural language description of the operation; The verification module is used to evaluate whether the operation is successfully executed based on the generated natural language description and the screen state sequence before and after the operation, and output a result containing verification information or error reasons; an information identification module, for extracting demand-related content from the target application based on the current user demand and the screen state after verification, if the evaluation operation is successfully executed; The user guidance module is used to integrate historical interaction information, current user needs and extracted content related to the needs, determine whether user interaction is required to refine the needs, and when user interaction is required to refine the needs, receive the refined demand information given by the user based on the guided interaction content, and then generate the next operation suggestion based on the updated user needs and historical operations.

[0060] See Figures 4 to 7 , is an example of the workflow of each module in a certain interactive step. Figure 4 An example of the workflow of the execution module in a certain interaction step; Figure 5 To verify the workflow example of the module in a certain interaction step, Figure 6 This is an example of the workflow of the information identification module in a certain interaction step. Figure 7 This is an example of the workflow of the user guidance module in a certain interaction step t in an embodiment of the present invention.

[0061] Specifically, the execution module, or the execution agent, is responsible for receiving current requirements, historical information, guidance suggestions, and possible error feedback, parsing, and performing specific operations on the application interface (such as clicking, sliding, entering text, etc.). The verification module, or the verification agent, is responsible for comparing the operation description of the execution module with the screen status before and after the operation, determining whether the operation is successfully executed, and outputting the verification result or the cause of the error. The information identification module, or the identification agent, is responsible for extracting information content related to user requirements from the current screen status of the application after the verification module confirms that the operation is successful. The user guidance module, or the guidance agent, is responsible for integrating information from all parties (requirements, history, extracted content), generating user-oriented guidance interaction content, receiving user feedback to update requirements, and generating next operation suggestions for the execution module.

[0062] The system further comprises: Memory module: Responsible for storing and managing status information throughout the entire interaction process, including user needs, action description history, screen state transitions, extracted content lists, etc., to achieve information sharing and status synchronization between different modules.

[0063] Screen representation module: responsible for obtaining and representing the interface state of the application, and can support multiple formats, such as view hierarchy-based (XML mode) or visual markup-based (SoM mode).

[0064] Since the system described in the second embodiment of the present invention is the system used to implement the multi-agent-based interactive instruction refinement and execution method described in the first embodiment of the present invention, those skilled in the art will be able to understand the specific structure and variations of the system based on the method described in the first embodiment of the present invention, and therefore will not be described in detail here. All systems used in the method described in the first embodiment of the present invention fall within the scope of protection of the present invention.

[0065] The methods and systems of the present invention have the following advantages: 1. Handling uncertain demands: Designed specifically for IIUN scenarios, they effectively handle situations where the user's initial intent is unclear. 2. Bidirectional information flow: Establishes an information feedback path from application to user (DA extracts information -> GA guides the user), which overcomes the shortcomings of traditional one-way execution. 3. Multi-agent collaboration: Through the collaborative work of multiple specialized agents (EA, VA, DA, GA) with clear division of labor, the robustness and accuracy of handling complex interactive tasks are improved. 4. Active guidance: The guiding agent can actively guide the user based on real-time information provided by the application, rather than passively waiting for user instructions, making the interaction more natural and efficient. 4. Improved usability in real-world scenarios: More closely aligned with the real-world user-mobile interaction model, this enhances the practical value of intelligent agents in complex tasks.

[0066] Example 3 Based on the same inventive concept, the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for interactive instruction refinement and execution based on multiple agents of embodiment one is implemented.

[0067] Since the computer-readable storage medium described in the third embodiment of the present invention is a computer device used to implement the multi-agent-based interactive instruction refinement and execution method described in the first embodiment of the present invention, those skilled in the art will be able to understand the specific structure and variations of the computer-readable storage medium based on the method described in the first embodiment of the present invention, and therefore will not be described in detail here. All computer-readable storage media used in the method of the first embodiment of the present invention fall within the scope of protection of the present invention.

[0068] Example 4 Based on the same inventive concept, the present invention further provides a computer device, including a memory 401, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in Embodiment 1 when executing the program.

[0069] Since the computer device described in the fourth embodiment of the present invention is the computer device used to implement the multi-agent-based interactive instruction refinement and execution method described in the first embodiment of the present invention, those skilled in the art will be able to understand the specific structure and variations of the computer device based on the method described in the first embodiment of the present invention, and therefore will not be described in detail here. All computer devices used in the method of the first embodiment of the present invention fall within the scope of protection of the present invention.

[0070] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, the present invention is intended to include such changes and modifications to the embodiments of the present invention if they fall within the scope of the claims and their equivalents.

Claims

1. A multi-agent-based interactive instruction refinement and execution method, characterized in that: include: Receive natural language instructions input by the user and perform preliminary analysis on the received instructions to obtain initial user needs; Based on the current understanding of user needs, perform an action on the target application and generate a natural language description of the action; Based on the generated natural language description and the screen state sequence before and after the operation, evaluate whether the operation is successfully executed and output a result containing verification information or error reasons; In the case where the evaluation operation is successfully executed, content related to the requirement is extracted from the target application according to the current user requirement and the screen status after verification; Based on historical interaction information, current user needs, and extracted content related to the needs, it is determined whether user interaction is required to refine the needs. If user interaction is required to refine the needs, the system receives the refined need information given by the user based on the guided interaction content, and then generates the next operation suggestion based on the updated user needs and historical operations.

2. The method for interactive instruction refinement and execution based on multi-agents according to claim 1, characterized in that: After receiving a natural language instruction input by a user and performing a preliminary analysis on the received instruction to obtain an initial user demand, the method further includes: The initial user demand obtained through analysis is stored in a memory module, wherein the memory module stores historical action information, current screen status, optional operation suggestions or error causes.

3. The method for interactive instruction refinement and execution based on multi-agents according to claim 2, characterized in that: Based on the current understanding of user needs, perform an action on the target application and generate a natural language description of the action, including: Based on the current user's needs, historical action information, current screen status, optional operation suggestions or error causes provided by the memory module, the operation decision function is called to determine and execute the operation on the application interface, and a natural language description of the operation is generated.

4. The method for interactive instruction refinement and execution based on multi-agents according to claim 1, characterized in that: Based on the generated natural language description and the screen state sequence before and after the operation, the system evaluates whether the operation is successfully executed and outputs a result containing verification information or error reasons, including: Receive the generated operation description and the screen state sequence before and after the operation, and call the verification function ,Evaluate whether the natural language description is successfully executed.

5. The method for interactive instruction refinement and execution based on multi-agents according to claim 1, characterized in that: If the evaluation operation is successfully executed, the content related to the requirement is extracted from the target application based on the current user requirement and the screen status after verification, including: In the case that the evaluation operation is successfully executed, the information extraction function is called according to the current user demand and the screen status after verification to extract the content related to the demand from the target application and store it in the memory module.

6. The method for interactive instruction refinement and execution based on multi-agents according to claim 1, characterized in that: When user interaction is required to refine requirements, the system receives the refined requirements information provided by the user based on the guided interaction content, and then generates the next step of operation suggestions based on the updated user requirements and historical operations, including: When user interaction is required to refine requirements, generate guidance interaction content and display it to the user, so that the user can give detailed requirements based on the guidance content; Call the demand update and suggestion generation function, integrate the refined demand with the user's original demand to obtain the updated user demand, and generate the next operation based on this, and update the updated user demand to the memory module.

7. The method for interactive instruction refinement and execution based on multi-agents according to claim 6, characterized in that: When user interaction is required to refine requirements, generate interactive content and display it to the user, specifically: in, Indicates the user's original needs. Represents the sequence of screen states before and after an operation. Indicates the screen state number traversed in this round of operation. Represents historical actions, Represents the extracted content sequence related to the requirements, Indicates that the output will be provided to the user. Indicates the generated guided interactive content. Indicates the current operation step; Call the demand update and suggestion generation function to integrate the refined demand with the current user's demand to obtain the updated user demand, specifically: in, Indicates that the output will be provided to the execution module, Indicates that users provide detailed requirements based on the guidance content. To meet the updated user needs, Provide suggestions for next steps.

8. A multi-agent based interactive instruction refinement and execution system, characterized by: include: The instruction receiving and parsing module is used to receive natural language instructions input by the user and perform preliminary parsing on the received instructions to obtain the initial user requirements; The execution module is used to perform operations on the target application based on the current understanding of user needs and generate a natural language description of the operation; The verification module is used to evaluate whether the operation is successfully executed based on the generated natural language description and the screen state sequence before and after the operation, and output a result containing verification information or error reasons; an information identification module, for extracting demand-related content from the target application based on the current user demand and the screen state after verification, if the evaluation operation is successfully executed; The user guidance module is used to integrate historical interaction information, current user needs and extracted content related to the needs, determine whether user interaction is required to refine the needs, and when user interaction is required to refine the needs, receive the refined demand information given by the user based on the guided interaction content, and then generate the next operation suggestion based on the updated user needs and historical operations.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, which, when executed by a processor, implements the multi-agent-based interactive instruction refinement and execution method as described in any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the multi-agent based interactive instruction refinement and execution method as described in any one of claims 1 to 7.

Images