Intelligent agent-based interaction method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide an interactive method and device based on an agent, electronic equipment and storage medium, and relate to the technical field of intelligent interaction. In response to a current user instruction, a current interface screenshot of a terminal is obtained. According to a retrieval library of a standard operation process and a user habit knowledge base, the current user instruction is rewritten to generate a target operation instruction and a target operation process. The retrieval library includes a mapping relationship between operation instructions and standard operation processes, and the user habit knowledge base includes preference information of a user. The target operation instruction, the target operation process and the current interface screenshot are input into an agent, so that the agent executes a task requested by the current user instruction. The present application can realize accurate alignment of an agent and human intention, so that the agent can highly understand the real intention of different users, and improve the ability of the agent to simulate human operation to complete a task and the degree of individualization.

Description

Technical Field

[0001] This application relates to the field of intelligent interaction technology, and in particular, to an interaction method, device, electronic device, and storage medium based on intelligent agents. Background Technology

[0002] A Graphical User Interface (GUI) agent is an artificial intelligence system driven by a Multimodal Large Language Model (MLLM). It can automatically reason and execute user interface (UI) interactions, simulating human user actions such as clicking, inputting, dragging, and reading interface information to complete tasks required by humans. With the enhancement of MLLM in decision-making, reasoning, and perception, GUI agents built on MLLM can now autonomously operate smart terminals such as mobile phones, computers, and cockpits according to user instructions. However, current GUI agents cannot fully understand the user's true intentions, cannot accurately simulate human operations to complete tasks according to the user's true intentions, and cannot be individually fine-tuned to customize the GUI agent for different users. Summary of the Invention

[0003] This application provides an interaction method, device, electronic device, and storage medium based on intelligent agents to solve the above-mentioned technical problems.

[0004] In a first aspect, embodiments of this application provide an interaction method based on an intelligent agent. The method includes: responding to a current user instruction and obtaining a screenshot of the current interface of the terminal; rewriting the current user instruction according to a pre-built standard operation procedure retrieval library and a user habit knowledge base to generate a target operation instruction and a target operation procedure, wherein the retrieval library includes a mapping relationship between operation instructions and standard operation procedures, and the user habit knowledge base includes user preference information; and inputting the target operation instruction, the target operation procedure, and the current interface screenshot into the intelligent agent, so that the intelligent agent executes the task requested by the current user instruction.

[0005] Secondly, embodiments of this application provide an interactive device based on an intelligent agent. The device includes: an information acquisition module, used to acquire a screenshot of the current interface of a terminal in response to a current user instruction; an information rewriting module, used to rewrite the current user instruction according to a pre-built standard operation procedure retrieval library and a user habit knowledge base, generating a target operation instruction and a target operation procedure, wherein the retrieval library includes a mapping relationship between operation instructions and standard operation procedures, and the user habit knowledge base includes user preference information; and an action execution module, used to input the target operation instruction, the target operation procedure, and the current interface screenshot into the intelligent agent, so that the intelligent agent executes the task requested by the current user instruction.

[0006] Thirdly, embodiments of this application provide an electronic device. The electronic device includes a memory and a processor, wherein the memory stores an application program that, when invoked by the processor, causes the processor to execute the method provided in embodiments of this application.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium stores program code, which, when invoked by a processor, causes the processor to execute the method provided in embodiments of this application.

[0008] This application provides an interaction method, device, electronic device, and storage medium based on an intelligent agent. According to a pre-built standard operating procedure (SOP) retrieval library and user habit knowledge base, the current user instruction is rewritten to generate target operation instructions and target operation flows. These target operation instructions and flows conform to the user's explicit intent (e.g., operation logic, step sequence) and implicit intent (e.g., user habits, personal preferences), thereby achieving precise alignment between the intelligent agent and human intent. This enables the intelligent agent to highly understand the true intent of different users, improving the agent's ability to simulate human operations and enhance its personalization. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating an embodiment of the agent-based interaction method provided in this application is shown. Figure 2A flowchart illustrating an agent-based interaction method provided in another embodiment of this application is shown. Figure 3 A framework diagram of an agent-based interaction method provided in an exemplary embodiment of this application is shown; Figure 4 A schematic diagram showing the experimental results of applying the method of this application to eight mobile GUI smart agents is illustrated. Figure 5 The diagram illustrates the experimental results of applying various demonstration learning methods to GUI agents. Figure 6 A schematic diagram of the single-step success rate score of the method of this application in experiments with different model parameter scales is shown; Figure 7 A schematic diagram of the type scores of the method of this application in experiments with different model parameter scales is shown; Figure 8 A schematic diagram of the intent alignment score of the method of this application in experiments with different model parameter scales is shown; Figure 9 This diagram illustrates the experimental results of this application after modifying some data on the publicly available mobile GUI intelligent agent evaluation benchmark OS-Kairos; Figure 10 This paper shows a structural block diagram of an agent-based interactive device according to an embodiment of the present application; Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0012] Existing work has, to some extent, enhanced the capabilities of GUI agents through human demonstration. Currently, demonstration-based learning methods for GUI agents focus solely on explicit human intent flows (e.g., operational logic, step sequences) to help the GUI agent learn how to simulate human actions to complete tasks. However, real-world user instructions are often ambiguous and user-specific. GUI agents cannot understand implicit human intent flows (e.g., personal preferences, user habits), leading to a misalignment between the GUI agent and the human user's intent. Consequently, the GUI agent cannot fully understand the user's true intent, cannot accurately simulate human actions according to the user's true intent to complete tasks, and cannot be individually fine-tuned for different users to customize the GUI agent for each user.

[0013] To address the aforementioned technical problems, this application provides an interaction method, device, electronic device, and storage medium based on an intelligent agent. The method of this application rewrites the current user instruction based on a pre-built standard operation procedure retrieval library and a user habit knowledge base, generating target operation instructions and target operation procedures. This ensures that the target operation instructions and target operation procedures conform to the user's explicit intentions (e.g., operation logic, step sequence) and implicit intentions (e.g., user habits, personal preferences), thereby achieving precise alignment between the intelligent agent and human intentions. This enables the intelligent agent to highly understand the true intentions of different users, improving the intelligent agent's ability to simulate human operations to complete tasks and enhancing its personalization.

[0014] The agent-based interaction method in this application can be applied to agent-based interaction devices or electronic devices. The agent-based interaction device can be deployed in an electronic device. An electronic device refers to a (terminal) device with Bluetooth connectivity, communication capabilities, computing power, and a display screen. Electronic devices can include, but are not limited to, smartphones, tablets, laptops, desktop computers, smart home appliances, wearable devices, in-vehicle computers, and virtual reality devices. Electronic devices can communicate with other electronic devices or servers via wired or wireless networks. A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), big data, and artificial intelligence platforms. The server can be used to provide services for applications running on the electronic device.

[0015] Please see Figure 1 , Figure 1 A flowchart illustrating an embodiment of the agent-based interaction method provided in this application is shown. Figure 1 As shown, the agent-based interaction method may include steps S110 to S130.

[0016] Step S110: Respond to the current user command and obtain a screenshot of the current interface of the terminal.

[0017] User instructions can include one or more tasks requested by a human to be performed by the terminal. A user can send user instructions to the terminal, causing the GUI agent within the terminal to simulate human actions and perform one or more tasks requested in the user instructions. As an example, the user instruction could be "Send a New Year's greeting to my leader." Upon receiving this instruction, the GUI agent would execute the following standard procedure: "① Open your phone, find and tap the WeChat application; ② Enter the WeChat interface and tap the search box; ③ Enter 'leader' in the search box; ④ Select 'leader' from the list of contacts; ⑤ Enter the New Year's greeting in the chat box: 'Wishing my leader a happy new year and continued success for the team!'; ⑥ Tap the send button," thus executing the user instruction's request to "send a New Year's greeting to my leader."

[0018] Screenshots are used to help the GUI agent understand user intent and the tasks requested by user commands, as well as to help the GUI agent know where to start executing the task from the standard operating procedure. For example, suppose the current user command is "Send a New Year's greeting to my leader," and the current screenshot is the "leader's" chat interface. Then the GUI agent can start executing from step ⑤, instead of starting from step ①.

[0019] Users can input user commands into the terminal via voice or text. Upon receiving a user command, the terminal can capture a screenshot of the current interface.

[0020] Step S120: Based on the pre-built standard operation procedure retrieval library and user habit knowledge base, rewrite the current user command to generate the target operation command and target operation procedure. The standard operation procedure retrieval library includes the mapping relationship between operation commands and standard operation procedures, and the user habit knowledge base includes user preference information.

[0021] The standard operating procedure (SOP) retrieval library can include multiple sets of operating instructions and SOPs, with each set having a one-to-one correspondence between the operating instructions and the SOP. A SOP typically includes multiple operations. Operating instructions can be stored in the SOP retrieval library as vectors. For example, an embedding model can be used. Operation instructions Encoding as latent representation Each set of operation instructions and standard operating procedures The mapping relationship between them is The data is stored in the retrieval database of the standard operating procedure.

[0022] In some embodiments, the most similar operation instruction to the current user instruction can be obtained from a standard operation procedure retrieval library, and the corresponding standard operation procedure can be obtained. Based on the most similar instruction, the corresponding standard operation procedure, the current user instruction, and the user habit knowledge base, a target operation instruction and a target operation procedure are generated.

[0023] In some embodiments, operation instructions in the retrieval library of standard operation procedures can be matched with the current user instruction for similarity, thereby achieving Retrieval-Augmented Generation (RAG). When an operation instruction has a similarity higher than a similarity threshold with the current user instruction, the operation instruction with the highest similarity to the current user instruction is selected as the most similar instruction. The similarity threshold can be preset according to the alignment requirements between the agent and human intent.

[0024] In some embodiments, the standard operating procedure retrieval library may include a mapping relationship between users, the user's operation instructions, and standard operating procedures, thereby improving the personalization of instruction rewriting and making the rewritten instructions more consistent with the user's true intent. Similarity matching may include: extracting all operation instructions of the current user from the standard operating procedure retrieval library, and calculating the similarity (e.g., cosine similarity) between each of the current user's operation instructions and the current user's instruction. When there is an operation instruction of the current user with a similarity higher than a similarity threshold, the operation instruction with the highest similarity to the current user's instruction is selected as the most similar instruction.

[0025] In some embodiments, if no operation instruction has a similarity higher than the similarity threshold with the current user instruction, the current user instruction can be directly used as the most similar instruction.

[0026] For example, an embedded model can be used. Encode the current user command q into a vector. Link vector l with all explicit intent stream records of the user. Similarity matching is performed to achieve RAG. When the similarity between vector l and operation instruction exceeds a similarity threshold τ, the system obtains the most similar instruction q' and its standard operation procedure p'. That is, if there exists... ,but: ;otherwise, .

[0027] In some embodiments, a standard operation procedure corresponding to the current user instruction can be determined based on the most similar instruction, the standard operation procedure corresponding to the most similar instruction, and the current user instruction. A target operation instruction and a target operation procedure are then generated based on the current user instruction, the standard operation procedure corresponding to the current user instruction, and a user habit knowledge base.

[0028] For example, the most similar command q', the standard operation flow p' corresponding to the most similar command, and the current user command q can be combined to form a few-shot prompt, which can then be input into the standard operation flow extractor E to obtain the standard operation flow corresponding to the current user command q. Few-shot prompting is a method that leverages a large language model to learn and process tasks from a small number of example samples. Then, the current user command q, the extracted standard operating procedure p, and the current user's... Habits Knowledge Base The common input query rewriter W generates target operation instructions that match the current user's personalized intent. and target operation process .

[0029] Step S130: Input the target operation instruction, target operation process, and current interface screenshot into the agent so that the agent can execute the task requested by the current user instruction.

[0030] For example, the rewritten query can Operating procedures The current screenshot s is input into the GUI agent F. The GUI agent F uses GUI proxy technology to process the rewritten query. Operating procedures Take a screenshot of the current interface and output the final action executed: This allows the system to execute tasks requested by the current user command. GUI proxy technology refers to techniques that enable automated interaction through a GUI.

[0031] Steps S110 to S130 have the following technical effects: Based on the pre-built standard operation process retrieval library and user habit knowledge base, the current user instruction is rewritten to generate target operation instructions and target operation processes, so that the target operation instructions and target operation processes conform to the user's explicit intentions (e.g., operation logic, step sequence) and implicit intentions (e.g., user habits, personal preferences), thereby achieving precise alignment between the intelligent agent and human intentions, enabling the intelligent agent to highly understand the true intentions of different users, and improving the intelligent agent's ability to simulate human operations to complete tasks and its personalization.

[0032] Please see Figure 2 , Figure 2 A flowchart illustrating an agent-based interaction method according to another embodiment of this application is shown. Figure 2 As shown, the agent-based interaction method may include steps S210 to S240.

[0033] Please see Figure 3 , Figure 3 A framework diagram of an agent-based interaction method provided in an exemplary embodiment of this application is shown. Figure 3 As shown, the agent-based interaction method in this embodiment includes the IFRAgent two-stage framework, namely, IFRAgent comprises two stages: intent stream extraction (including step S210) and deployment (including steps S220 to S240). Figure 3 As shown in the upper part, in the intent flow extraction stage (including step S210), IFRAgent, based on human demonstration data intent flow recognition technology, constructs a retrieval library of SOPs by analyzing the explicit intent flow (the correspondence between operation instructions and SOPs) of human demonstration data. Simultaneously, it constructs a user-level habit knowledge base by analyzing the implicit intent flow (user preference information) of human demonstration data. For example... Figure 3 As shown in the lower half, during the deployment phase (including steps S220 to S240), IFRAgent, as a plug-and-play module, combines a SOP extractor and instruction rewriter with RAG functionality. Based on the preliminary intent flow analysis results, it rewrites ambiguous user instructions into personalized instructions and customized SOPs that conform to user characteristics, thereby achieving precise alignment between the mobile agent and human intent. The IFRAgent framework, this plug-and-play system, proposes a GUI agent user instruction personalization rewriting technology by integrating explicit and implicit intent flows from human demonstration data, significantly improving the task completion capability of the mobile agent and the performance of user-specific intent alignment.

[0034] Step S210: Based on human demonstration data, construct a retrieval library for standard operating procedures and a user habit knowledge base, wherein the human demonstration data includes operating instructions and a sequence of screenshots of user actions in response to the operating instructions.

[0035] Human demonstration data may include, but is not limited to, operation commands, sequences of screenshots from multiple users, and sequences of screenshots showing user actions in response to operation commands. Specifically, a sequence of screenshots includes multiple user interface images; that is, a sequence of screenshots refers to a set of user interface images captured during the execution of a set of user actions in a specific order.

[0036] Human demonstration data can be sourced from the open-source Mobile Intention Alignment Rate (MobileIAR) dataset. MobileIAR is a dataset designed to evaluate the alignment between GUI agents and human intentions, specifically tailored for assessing the alignment rate between mobile agents and human intents. The MobileIAR dataset encompasses 945 user commands from 16 applications, covering seven major categories of everyday scenarios. It not only provides standard action sequences that conform to human intentions but also includes complete standards for real-world operation chains, allowing for a comprehensive evaluation of the alignment between mobile agents and human intentions. The MobileIAR dataset includes both user intention-aligned actions and real-world operation sequences, reflecting not only traditional metrics such as task completion rate but also quantifying the degree of matching between mobile agents and user intentions. It establishes, for the first time, a user-specific intent alignment benchmark for the mobile agent domain.

[0037] For example, such as Figure 3 As shown, in the intent stream extraction phase, IFRAgent extracts the user's explicit and implicit intent streams by collecting and analyzing human demonstration data. For the user set... Each user in We first collect a set of operation instructions. Human demonstration data was used to initialize an empty user-level habit knowledge base. Each operation instruction Attached is a sequence of screenshots provided by the user. .

[0038] In some embodiments, an explicit intent stream can be extracted from a sequence of operation screenshots. The explicit intent stream includes operation instructions and corresponding standard operation procedures. Based on the explicit intent stream, a retrieval library of standard operation procedures is constructed. For example, given a tuple... It can be done through explicit intent stream proxy (The publicly available explicit intent flow extraction model) is processed to extract standard operating procedures: Meanwhile, the operation command qⱼ is transmitted through the embedded model. Encoding as latent representation ,Right now: Characterization It will be stored in the retrieval database of the standard operating procedure for users. Retrieved and used during the deployment phase.

[0039] The results of extracting explicit intent streams include: from a user The screenshots show the operation commands and their corresponding standard operation procedures obtained from the sequence of operations. For ease of understanding, Examples 1 to 3 are provided to illustrate the operation commands and their corresponding standard operation procedures: Example 1: The instruction is "Send a New Year's greeting to my leader". The standard operation process for this instruction includes: "Open your phone, find and click the WeChat application; enter the WeChat interface and click the search box; enter 'leader' in the search box; select 'leader' from the contacts that appear; enter the New Year's greeting in the chat box: 'Wishing my leader a happy new year and the team continued success!'; click the send button."

[0040] Example 2: The operation instruction is "Tell my leader that I will attend the meeting at 9 pm tonight on time". The standard operation process for this operation instruction includes "Open the WeChat application; click the search box; enter leader and search; select the contact leader; enter "Received, I will attend the meeting at 9 pm tonight on time" in the input box; click the send button to send the message.

[0041] Example 3: The operation instruction is "Tell my wife I won't be home for dinner tonight". The standard operation procedure for this instruction includes "Open your phone; open the WeChat application; open the search icon in the upper right corner; enter 'wife' to search; open the contact 'wife' in the search results; enter 'I'm working overtime tonight and won't be home for dinner, love you' in the chat box; open the send button".

[0042] In some embodiments, an implicit intent stream, including user preference information, can be extracted from a sequence of operation screenshots; a user habit knowledge base can then be constructed based on the implicit intent stream. For example, a binary tuple... Through implicit intent flow proxy (The implicit intent flow extraction model) is used for processing, and the agent incrementally updates the habitual knowledge base: Among them, implicit intent stream proxy Potential behavioral patterns can be learned from interaction sequences.

[0043] The results of implicit intent stream extraction include: from a certain user The software usage preferences, domain behavior preferences, and software behavior preferences are obtained from the sequence of screenshots. For example, a user's software usage preferences may include: using app A to order food, using app B for map navigation, using app C to chat, using app D to watch videos, using app E to shop, using app F to search, and using app G to listen to music. For example, a user's domain behavior preferences may include: when ordering food, the user prefers coconut latte, standard strength, and regular ice; frequently chooses spicy hot pot; prioritizes speed; searches for nearby roast duck restaurants; and orders delivery for 9 AM the next day. When traveling, the user prefers to search for nearby destinations based on location.

[0044] like Figure 3 As shown in the upper part, the dual-channel processing framework in the intent stream extraction stage traverses the set of operation instructions. All operational instructions are processed until all human demonstration data has been processed, ultimately generating a complete habit knowledge base for each user. and standard operating procedure retrieval database .

[0045] Step S220: Respond to the current user command and obtain a screenshot of the current interface of the terminal.

[0046] Step S230: Based on the pre-built standard operation procedure retrieval library and user habit knowledge base, rewrite the current user command to generate the target operation command and target operation procedure. The retrieval library includes the mapping relationship between the operation command and the standard operation procedure, and the user habit knowledge base includes the user's preference information.

[0047] Step S240: Input the target operation instruction, target operation process, and current interface screenshot into the agent so that the agent can execute the task requested by the current user instruction.

[0048] For detailed explanations of steps S220 to S240, please refer to steps S110 to S130, which will not be repeated here.

[0049] Please see Figure 4 , Figure 4 The diagram illustrates the experimental results of applying the method of this application to eight mobile GUI agents. The experimental results show that IFRAgent outperforms the baseline method by an average of 6.79% (a relative improvement of 32.06%) in human intent alignment rate and by an average of 5.30% (a relative improvement of 26.34%) in step completion rate.

[0050] Please see Figure 5 , Figure 5The diagram illustrates the experimental results of applying various demonstrative learning methods to GUI agents. The method presented in this application leads the field compared to similar demonstrative learning methods, achieving results ranging from being on par with to exceeding similar methods by 10% on various metrics.

[0051] Please see Figures 6 to 8 , Figures 6 to 8 The diagram illustrates various indices of the method described in this application under different model parameter scales. For example... Figures 6 to 8 As shown, this application can improve the performance by 5% to 20% in experiments with different model parameter scales.

[0052] Please see Figure 9 , Figure 9 This diagram illustrates the experimental results of this application after modifying some data on the publicly available mobile GUI intelligent agent benchmark OS-Kairos. OS-Kairos is a novel GUI intelligent agent system with adaptive interaction capabilities. This application re-aligned some data with the data intent standard on the publicly available mobile GUI intelligent agent benchmark OS-Kairos, and still achieved improvements in various indicators relative to the baseline level.

[0053] Through extensive experiments on different mobile agents, this invention demonstrates that IFRAgent achieves an average improvement of 6.79% (relative improvement of 32.06%) in intent alignment rate and an average improvement of 5.30% (relative improvement of 26.34%) in step success rate compared to the baseline method.

[0054] Steps S210 to S240 have the following technical effects: (1) Intent flow recognition technology based on human demonstration actions: Based on human demonstration data, a retrieval library of standard operating procedures and a user habit knowledge base are constructed. This can simultaneously provide the intelligent agent with explicit intent flow (i.e., the correspondence between operation instructions and standard operating procedures) and implicit intent flow (i.e., user preference information), thereby improving the personalization and accuracy of rewriting user instructions.

[0055] (2) Based on the pre-built standard operation process retrieval library and user habit knowledge base, the current user instruction is rewritten to generate target operation instructions and target operation process, so that the target operation instructions and target operation process conform to the user's explicit intention (e.g., operation logic, step sequence) and implicit intention (e.g., user habits, personal preferences), thereby achieving accurate alignment between the intelligent agent and human intention, enabling the intelligent agent to highly understand the real intentions of different users, and improving the intelligent agent's ability to simulate human operation to complete tasks and its personalization.

[0056] Please see Figure 10 , Figure 10 A structural block diagram of an agent-based interactive device according to an embodiment of this application is shown. Figure 10 As shown, the agent-based interactive device 100 may include an information acquisition module 110, an information rewriting module 120, and an action execution module 130.

[0057] The information acquisition module 110 is used to: respond to the current user's command and acquire a screenshot of the current interface of the terminal.

[0058] The information rewriting module 120 is used to: rewrite the current user instructions based on the pre-built standard operation procedure retrieval library and user habit knowledge base, and generate target operation instructions and target operation procedures. The standard operation procedure retrieval library includes the mapping relationship between operation instructions and standard operation procedures, and the user habit knowledge base includes user preference information.

[0059] The action execution module 130 is used to input the target operation instruction, the target operation process, and the current interface screenshot into the intelligent agent so that the intelligent agent can execute the task requested by the current user instruction.

[0060] In some embodiments, the information rewriting module 120 is further configured to: obtain the operation instruction most similar to the current user instruction from the standard operation procedure retrieval library as the most similar instruction, obtain the standard operation procedure corresponding to the most similar instruction; and generate a target operation instruction and a target operation procedure based on the most similar instruction, the standard operation procedure corresponding to the most similar instruction, the current user instruction, and the user habit knowledge base.

[0061] In some embodiments, the information rewriting module 120 is further configured to: determine the standard operation procedure corresponding to the current user instruction based on the most similar instruction, the standard operation procedure corresponding to the most similar instruction, and the current user instruction; and generate a target operation instruction and a target operation procedure based on the current user instruction, the standard operation procedure corresponding to the current user instruction, and the user habit knowledge base.

[0062] In some embodiments, the information rewriting module 120 is further configured to: perform similarity matching between the operation instructions in the retrieval library of the standard operation procedure and the current user instruction; when there is an operation instruction whose similarity to the current user instruction is higher than the similarity threshold, obtain the operation instruction with the highest similarity to the current user instruction as the most similar instruction.

[0063] In some embodiments, the agent-based interaction device 100 may further include a database construction module. The database construction module is used to: before acquiring the current user command and the current interface screenshot, construct a retrieval library of standard operating procedures and a user habit knowledge base based on human demonstration data, wherein the human demonstration data includes operating commands and a sequence of screenshots showing user actions in response to the operating commands.

[0064] In some embodiments, the database construction module is further configured to: extract an explicit intent stream from the sequence of operation screenshots, the explicit intent stream including operation instructions and standard operation procedures corresponding to the operation instructions; and construct a retrieval library of standard operation procedures based on the explicit intent stream.

[0065] In some embodiments, the database construction module is further configured to: extract an implicit intent stream from the sequence of operation screenshots, the implicit intent stream including user preference information; and construct a user habit knowledge base based on the implicit intent stream.

[0066] Those skilled in the art will clearly understand that the agent-based interaction device 100 provided in the embodiments of this application can implement the agent-based interaction method provided in the embodiments of this application. The specific working process of the above-mentioned device and modules can be found in the process corresponding to the agent-based interaction method in the embodiments of this application, and will not be repeated here.

[0067] In the embodiments provided in this application, the coupling, direct coupling, or communication connection between the modules shown or discussed may be indirect coupling or communication coupling through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms. The embodiments of this application do not limit this.

[0068] Furthermore, the functional modules in the embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules, and this application embodiment does not impose any restrictions on this.

[0069] Please see Figure 11 , Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device 200 may include a memory 210 and a processor 220. The memory 210 stores an application program, which, when invoked by the processor 220, causes the processor 220 to execute the agent-based interaction method described above in the embodiments of this application.

[0070] Processor 220 may include one or more processing cores. Processor 220 uses various interfaces and lines to connect to various parts of the entire electronic device 200, and is used to run or execute instructions, programs, code sets or instruction sets stored in memory 210, call and run or execute data stored in memory 210, and perform various functions of electronic device 200 and process data.

[0071] Optionally, the processor 220 can be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 220 can integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications. The GPU is responsible for rendering and drawing the displayed content. The modem handles wireless communication. It is understood that the modem can also be implemented separately as a communication chip, without being integrated into the processor 220.

[0072] The memory 210 may include random access memory (RAM) or read-only memory (ROM). The memory 210 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 210 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may store data created by the electronic device 200 during use.

[0073] This application also provides a computer-readable storage medium. The computer-readable storage medium stores program code, which, when invoked by a processor, causes the processor to execute the agent-based interaction method described above in this application.

[0074] Computer-readable storage media may include, but are not limited to, flash memory, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), hard disks, or ROM and other electronic storage devices. Optionally, computer-readable storage media may include non-volatile computer-readable storage medium (Non-TCRSM).

[0075] The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed in an appropriate form.

[0076] In summary, the embodiments of this application provide an interaction method, device, electronic device, and storage medium based on an intelligent agent. Based on a pre-built retrieval library of standard operating procedures and a user habit knowledge base, the current user instruction is rewritten to generate target operating instructions and target operating procedures. This ensures that the target operating instructions and target operating procedures conform to the user's explicit intent (e.g., operating logic, step sequence) and implicit intent (e.g., user habits, personal preferences), thereby achieving precise alignment between the intelligent agent and human intent. This enables the intelligent agent to highly understand the true intent of different users, improving the agent's ability to simulate human operations and enhance its personalization.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An agent-based interaction method, characterized in that, include: In response to the current user command, obtain a screenshot of the current interface of the terminal; Based on a pre-built standard operation procedure retrieval library and a user habit knowledge base, the current user instruction is rewritten to generate a target operation instruction and a target operation procedure. The retrieval library includes a mapping relationship between operation instructions and standard operation procedures, and the user habit knowledge base includes user preference information. The target operation instruction, the target operation process, and the current interface screenshot are input into the intelligent agent so that the intelligent agent can execute the task requested by the current user instruction. The step of rewriting the current user instruction based on a pre-built standard operation procedure retrieval library and user habit knowledge base to generate target operation instructions and target operation procedures includes: From the standard operation procedure retrieval library, obtain the operation instruction most similar to the current user instruction as the most similar instruction, and obtain the standard operation procedure corresponding to the most similar instruction; A few-sample prompt is constructed based on the most similar instruction, the standard operation procedure corresponding to the most similar instruction, and the current user instruction; and the standard operation procedure corresponding to the current user instruction is extracted based on the few-sample prompt. Based on the current user instruction, the standard operation procedure corresponding to the current user instruction, and the user habit knowledge base, generate the target operation instruction and the target operation procedure.

2. The method according to claim 1, characterized in that, The step of retrieving the operation instruction most similar to the current user instruction from the standard operation procedure retrieval library includes: The operation instructions in the standard operation procedure search library are matched with the current user instructions based on similarity. If an operation instruction has a similarity higher than the current user instruction, the operation instruction with the highest similarity to the current user instruction is selected as the most similar instruction.

3. The method according to claim 1 or 2, characterized in that, Before obtaining the current user command and the current interface screenshot, the method further includes: Based on human demonstration data, a retrieval library for standard operating procedures and a user habit knowledge base are constructed. The human demonstration data includes operating instructions and a sequence of screenshots of user actions in response to the operating instructions.

4. The method according to claim 3, characterized in that, Based on human demonstration data, a retrieval library for standard operating procedures was constructed, including: From the sequence of operation screenshots, an explicit intent stream is extracted, which includes operation instructions and standard operation procedures corresponding to the operation instructions. Based on the explicit intent flow, a retrieval library for standard operating procedures is constructed.

5. The method according to claim 3, characterized in that, Based on human demonstration data, a user habit knowledge base is constructed, including: From the sequence of operation screenshots, an implicit intent stream is extracted, which includes the user's preference information; Based on the implicit intent flow, a user habit knowledge base is constructed.

6. An interactive device based on an intelligent agent, characterized in that, include: The information acquisition module is used to acquire a screenshot of the current interface of the terminal in response to the current user command; The information rewriting module is used to rewrite the current user instruction based on a pre-built standard operation procedure retrieval library and a user habit knowledge base, to generate a target operation instruction and a target operation procedure. The retrieval library includes a mapping relationship between operation instructions and standard operation procedures, and the user habit knowledge base includes user preference information. An action execution module is used to input the target operation instruction, the target operation process, and the current interface screenshot into the intelligent agent, so that the intelligent agent can execute the task requested by the current user instruction. The step of rewriting the current user instruction based on a pre-built standard operation procedure retrieval library and user habit knowledge base to generate target operation instructions and target operation procedures includes: From the standard operation procedure retrieval library, obtain the operation instruction most similar to the current user instruction as the most similar instruction, and obtain the standard operation procedure corresponding to the most similar instruction; A few-sample prompt is constructed based on the most similar instruction, the standard operation procedure corresponding to the most similar instruction, and the current user instruction; and the standard operation procedure corresponding to the current user instruction is extracted based on the few-sample prompt. Based on the current user instruction, the standard operation procedure corresponding to the current user instruction, and the user habit knowledge base, generate the target operation instruction and the target operation procedure.

7. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores an application program that, when invoked by the processor, causes the processor to perform the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that, when invoked by a processor, causes the processor to perform the method as described in any one of claims 1-5.

Images