Interaction method and device based on intelligent agent, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an interaction method and device based on an intelligent agent, electronic equipment and a storage medium, and relates to the technical field of intelligent interaction. The method comprises the steps of obtaining a current interface screenshot of a terminal in response to a current user instruction; according to a pre-constructed standard operation process retrieval library and a pre-constructed user habit knowledge base, the current user instruction is rewritten, a target operation instruction and a target operation process are generated, the retrieval library comprises a mapping relation between the operation instruction and the standard operation process, and the user habit knowledge base comprises preference information of a user; and inputting the target operation instruction, the target operation process and the current interface screenshot into an intelligent agent, so that the intelligent agent executes a task requested by the current user instruction. According to the method, the intelligent agent and the human intention can be accurately aligned, so that the intelligent agent can highly understand real intentions of different users, and the ability and the individuation degree of the intelligent agent for simulating human operation to complete tasks are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of intelligent interaction, in particular, to an interaction method and device based on an agent, an electronic device and a storage medium. BACKGROUND

[0002] A graphical user interface (GUI) agent is an artificial intelligence system driven by a multimodal large language model (MLLM), which can automatically infer and perform user interface (UI) interaction, simulate human operations such as clicking, inputting, dragging, and reading interface information, and thus complete human required work tasks. With the enhancement of MLLM in decision-making, reasoning, and perception, the GUI agent based on MLLM can independently operate smart terminals such as mobile phones, computers, and cockpits according to user instructions. However, the current GUI agent cannot fully understand the real intention of the user, cannot accurately simulate human operations according to the real intention of the user to complete tasks, and cannot individually fine-tune different users to customize GUI agents for different users. SUMMARY

[0003] Embodiments of the present application provide an interaction method and device based on an agent, an electronic device and a storage medium to solve the above technical problems.

[0004] In a first aspect, embodiments of the present application provide an interaction method based on an agent. The method comprises: in response to a current user instruction, acquiring a current interface screenshot of a terminal; rewriting the current user instruction according to a retrieval library of a standard operation process and a user habit knowledge base pre-constructed, to generate a target operation instruction and a target operation process, wherein the retrieval library comprises a mapping relationship between operation instructions and standard operation processes, and the user habit knowledge base comprises preference information of a user; inputting the target operation instruction, the target operation process, and the current interface screenshot into an agent, to make the agent execute a task requested by the current user instruction.

[0005] In a second aspect, an embodiment of the present application provides an interactive device based on an agent. The device comprises: an information acquisition module configured to acquire a current interface screenshot of a terminal in response to a current user instruction; an information rewriting module configured to rewrite the current user instruction according to a retrieval library of a standard operating procedure and a user habit knowledge base, to generate a target operating instruction and a target operating procedure, wherein the retrieval library comprises a mapping relationship between an operating instruction and a standard operating procedure, and the user habit knowledge base comprises preference information of a user; and an action execution module configured to input the target operating instruction, the target operating procedure and the current interface screenshot into an agent, to enable the agent to perform a task requested by the current user instruction.

[0006] In a third aspect, an embodiment of the present application provides an electronic device. The electronic device comprises a memory and a processor, and the memory stores an application program, which is configured to enable the processor to perform a method provided by an embodiment of the present application when the application program is invoked by the processor.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a program code, which is configured to enable a processor to perform a method provided by an embodiment of the present application when the program code is invoked by the processor.

[0008] An embodiment of the present application provides an interactive method and device based on an agent, a electronic device and a storage medium. The current user instruction is rewritten according to a retrieval library of a standard operating procedure (SOP) and a user habit knowledge base, to generate a target operating instruction and a target operating procedure, so that the target operating instruction and the target operating procedure conform to explicit intention (for example, operation logic and step sequence) and implicit intention (for example, user habit and personal preference) of a user, thereby realizing accurate alignment of the agent and the human intention, enabling the agent to highly understand real intention of different users, and improving the ability of the agent to simulate human operation to complete a task and the degree of individualization. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0010] Figure 1 A flowchart of an interactive method based on an agent provided by an embodiment of the present application is shown; Figure 2A flowchart of an embodiment of the method is shown in FIG. 1. Figure 3 A framework of an embodiment of the method is shown in FIG. 2. Figure 4 A diagram of experimental results of experiments of the method applied to eight mobile phone GUI agents is shown in FIG. 3. Figure 5 A diagram of experimental results of experiments of various exemplary learning methods applied to GUI agents is shown in FIG. 4. Figure 6 A diagram of single-step success rate scores of experiments of the method in different model parameter scales is shown in FIG. 5. Figure 7 A diagram of type scores of experiments of the method in different model parameter scales is shown in FIG. 6. Figure 8 A diagram of intention alignment rate scores of experiments of the method in different model parameter scales is shown in FIG. 7. Figure 9 A diagram of experimental results of experiments of the method after modifying part of data of a public mobile phone GUI agent evaluation benchmark OS-Kairos is shown in FIG. 8. Figure 10 A block diagram of an embodiment of the apparatus is shown in FIG. 9. Figure 11 A block diagram of an embodiment of the electronic device is shown in FIG. 10. DETAILED DESCRIPTION

[0011] In order to make persons skilled in the art better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0012] Existing work can enhance the ability of a GUI agent to some extent through human demonstration. Currently, the demonstration learning method for a GUI agent only focuses on the explicit intention flow of a human (for example, operation logic, step sequence) to help the GUI agent learn how to simulate human operation to complete a task. However, the user instruction in a real scene often has ambiguity and user specificity, and the GUI agent cannot understand the implicit intention flow of the human (for example, personal preference, user habit), resulting in that the intention between the GUI agent and the human user cannot be aligned, the GUI agent cannot fully understand the real intention of the user, cannot simulate human operation according to the real intention of the user to complete the task, and cannot individually fine-tune for different users to customize the GUI agent for different users.

[0013] To solve the above technical problems, the application provides an agent-based interaction method and device, electronic equipment and storage medium. The application method rewrites the current user instruction according to the pre-constructed standard operation process search library and user habit knowledge base, generates target operation instruction and target operation process, so that the target operation instruction and target operation process meet the explicit intention (for example, operation logic, step sequence) and implicit intention (for example, user habit, personal preference) of the user, thereby realizing the accurate alignment of the agent and the human intention, enabling the agent to highly understand the real intention of different users, and improving the ability and individualization degree of the agent to simulate human operation to complete the task.

[0014] The agent-based interaction method in the application can be applied to an agent-based interaction device or electronic equipment. The agent-based interaction device can be deployed in the electronic equipment. The electronic equipment refers to a (terminal) device with Bluetooth connection function, communication capability, computing capability and display screen. The electronic equipment can include but is not limited to smart phones, tablet computers, notebook computers, desktop computers, smart home appliances, wearable devices, vehicle-mounted computers, virtual reality devices and the like. The electronic equipment can communicate with other electronic equipment or servers through wired or wireless network. The server can be a stand-alone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network (Content Delivery Network, CDN), big data and artificial intelligence platform and other basic cloud computing services. The server can be used to provide services for the application running on the electronic equipment.

[0015] Please refer to Figure 1 , Figure 1 The flowchart of the agent-based interaction method provided by an embodiment of the application is shown. As Figure 1 shown, the agent-based interaction method can include steps S110 to S130.

[0016] Step S110: In response to the current user instruction, the current interface screenshot of the terminal is obtained.

[0017] The user instruction can include one or more tasks requested by the user to be performed by the terminal. The user can send the user instruction to the terminal so that the GUI agent in the terminal simulates human operation to perform one or more tasks requested in the user instruction. As an example, the user instruction can be "send a new year's greetings to my leader", and the GUI agent will perform the following standard operation procedure after receiving the user instruction: "① open the mobile phone, find and click the WeChat application; ② enter the WeChat interface and click the search box; ③ enter "leader" in the search box; ④ select "leader" from the contacts; ⑤ enter the new year's greetings content "Wish the leader a happy new year and may the team achieve even greater success!" in the chat box; ⑥ click the send button", thereby performing the task of "sending a new year's greetings to my leader" requested by the user instruction.

[0018] The interface screenshot is used to help the GUI agent understand the user's intention and the task requested by the user instruction, and to help the GUI agent understand where to start performing the task from the standard operation procedure. For example, assuming that the current user instruction is "send a new year's greetings to my leader" and the current interface screenshot is the chat interface of "leader", the GUI agent can start from step ⑤ without having to start from step ①.

[0019] The user can input the user instruction in the form of voice or text into the terminal, and the terminal can obtain the current interface screenshot when receiving the user instruction.

[0020] Step S120: rewriting the current user instruction according to the pre-constructed retrieval library of standard operation procedures and the user habit knowledge base, generating target operation instructions and target operation procedures, wherein the retrieval library of standard operation procedures includes the mapping relationship between operation instructions and standard operation procedures, and the user habit knowledge base includes the user's preference information.

[0021] The retrieval library of standard operation procedures can include multiple groups of operation instructions and standard operation procedures, each group having a one-to-one correspondence between operation instructions and standard operation procedures. The standard operation procedure can generally include multiple operations. The operation instruction can be stored in the retrieval library of standard operation procedures in the form of a vector. As an example, an embedding model The operation instruction is encoded into a latent representation Each group of operation instructions and standard operation procedures The mapping relationship between them is stored in the retrieval library of standard operation procedures in the form of .

[0022] In some embodiments, the most similar instruction can be obtained from the retrieval library of standard operation procedures as the instruction most similar to the current user instruction, and the standard operation procedure corresponding to the most similar instruction can be obtained. The target operation instruction and the target operation procedure can be generated according to the most similar instruction, the standard operation procedure corresponding to the most similar instruction, the current user instruction, and the user habit knowledge base.

[0023] In some embodiments, the operation instructions in the retrieval library of standard operation procedures can be matched with the current user instruction respectively to achieve retrieval-augmented generation (RAG). When the similarity between the operation instruction and the current user instruction is higher than a similarity threshold, the operation instruction with the highest similarity to the current user instruction can be obtained as the most similar instruction. The similarity threshold can be set in advance according to the alignment rate requirement of the intelligent agent and the human intention.

[0024] In some embodiments, the retrieval library of standard operation procedures can include the mapping relationship between the user, the operation instruction of the user, and the standard operation procedure, thereby improving the individuality of instruction rewriting and making the rewritten instruction more consistent with the real intention of the user. The similarity matching can include extracting all operation instructions of the current user in the retrieval library of standard operation procedures, and calculating the similarity (such as cosine similarity) between each of the operation instructions of the current user and the current user instruction respectively. When the similarity between the operation instruction of the current user and the current user instruction is higher than the similarity threshold, the operation instruction with the highest similarity to the current user instruction can be obtained as the most similar instruction.

[0025] In some embodiments, when the similarity between the operation instruction and the current user instruction is not higher than the similarity threshold, the current user instruction can be directly obtained as the most similar instruction.

[0026] For example, an embedding model can be used The current user instruction q is encoded into a vector The vector l is matched with all explicit intention flow records of the user The similarity is matched to achieve RAG. When the similarity between the vector l and the operation instruction exceeds the similarity threshold τ, the system obtains the most similar instruction q' and its standard operation procedure p'. That is, if there is Then: Otherwise, .

[0027] In some embodiments, a standard operation procedure corresponding to the current user instruction can be determined based on the most similar instruction, the standard operation procedure corresponding to the most similar instruction, and the current user instruction. A target operation instruction and a target operation procedure are then generated based on the current user instruction, the standard operation procedure corresponding to the current user instruction, and a user habit knowledge base.

[0028] For example, the most similar command q', the standard operation flow p' corresponding to the most similar command, and the current user command q can be combined to form a few-shot prompt, which can then be input into the standard operation flow extractor E to obtain the standard operation flow corresponding to the current user command q. Few-shot prompting is a method that leverages a large language model to learn and process tasks from a small number of example samples. Then, the current user command q, the extracted standard operating procedure p, and the current user's... Habits Knowledge Base The common input query rewriter W generates target operation instructions that match the current user's personalized intent. and target operation process .

[0029] Step S130: Input the target operation instruction, target operation process, and current interface screenshot into the agent so that the agent can execute the task requested by the current user instruction.

[0030] For example, the rewritten query can Operating procedures The current screenshot s is used as input for the GUI agent F. The GUI agent F uses GUI proxy technology to process the rewritten query. Operating procedures Take a screenshot of the current interface and output the final action executed: This allows the system to execute tasks requested by the current user command. GUI proxy technology refers to techniques that enable automated interaction through a GUI.

[0031] Steps S110 to S130 have the following technical effects: Based on the pre-built standard operation process retrieval library and user habit knowledge base, the current user instruction is rewritten to generate target operation instructions and target operation processes, so that the target operation instructions and target operation processes conform to the user's explicit intentions (e.g., operation logic, step sequence) and implicit intentions (e.g., user habits, personal preferences), thereby achieving precise alignment between the intelligent agent and human intentions, enabling the intelligent agent to highly understand the true intentions of different users, and improving the intelligent agent's ability to simulate human operations to complete tasks and its personalization.

[0032] Please seeFigure 2 , Figure 2 A flowchart of an embodiment of the agent-based interaction method is shown. As shown in Figure 2 , the agent-based interaction method can include steps S210 to S240.

[0033] Referring to Figure 3 , Figure 3 A framework diagram of an embodiment of the agent-based interaction method is shown. As shown in Figure 3 , the agent-based interaction method in this embodiment includes an IFRAgent two-stage framework, i.e., IFRAgent includes two stages of intent flow extraction (including step S210) and deployment (including steps S220 to S240). As shown in the upper half of Figure 3 , in the intent flow extraction stage (including step S210), IFRAgent, based on the intent flow recognition technology of human demonstration data, constructs a retrieval library of SOPs by analyzing the explicit intent flow (the correspondence between operation instructions and SOPs) of human demonstration data, and at the same time, constructs a user-level habit knowledge base by analyzing the implicit intent flow (user preference information) of human demonstration data. As shown in the lower half of Figure 3 , in the deployment stage (including steps S220 to S240), IFRAgent, as a plug-and-play module, combines a SOP extractor with a command rewriter with RAG function, and based on the previous intent flow analysis results, rewrites user fuzzy instructions into personalized instructions and customized SOPs conforming to user characteristics, so as to realize the precise alignment of mobile agents and human intent. The IFRAgent framework, this plug-and-play system, by fusing the explicit and implicit intent flow in human demonstration data, proposes a GUI agent user instruction personalized rewriting technology, which significantly improves the task completion ability of mobile agents and the user-specific intent alignment performance.

[0034] Step S210: Constructing a retrieval library of standard operation procedures and a user habit knowledge base according to human demonstration data, wherein the human demonstration data includes operation instructions and operation screenshot sequences made by users for the operation instructions.

[0035] The human demonstration data can include but is not limited to operation instructions, operation screenshot sequences of multiple users, and operation screenshot sequences made by users for the operation instructions. The operation screenshot sequence includes multiple interface screenshots, i.e., the operation screenshot sequence refers to a set of interface screenshots in the execution process of a group of user operations in a specific order.

[0036] Human demonstration data can be sourced from the open-source Mobile Intention Alignment Rate (MobileIAR) dataset. MobileIAR is a dataset designed to evaluate the alignment between GUI agents and human intentions, specifically tailored for assessing the alignment rate between mobile agents and human intents. The MobileIAR dataset encompasses 945 user commands from 16 applications, covering seven major categories of everyday scenarios. It not only provides standard action sequences that conform to human intentions but also includes complete standards for real-world operation chains, allowing for a comprehensive evaluation of the alignment between mobile agents and human intentions. The MobileIAR dataset includes both user intention-aligned actions and real-world operation sequences, reflecting not only traditional metrics such as task completion rate but also quantifying the degree of matching between mobile agents and user intentions. It establishes, for the first time, a user-specific intent alignment benchmark for the mobile agent domain.

[0037] For example, such as Figure 3 As shown, in the intent stream extraction phase, IFRAgent extracts the user's explicit and implicit intent streams by collecting and analyzing human demonstration data. For the user set... Each user in We first collect a set of operation instructions. Human demonstration data was used to initialize an empty user-level habit knowledge base. Each operation instruction Attached is a sequence of screenshots provided by the user. .

[0038] In some embodiments, an explicit intent stream can be extracted from a sequence of operation screenshots. The explicit intent stream includes operation instructions and corresponding standard operation procedures. Based on the explicit intent stream, a retrieval library of standard operation procedures is constructed. For example, given a tuple... It can be done through explicit intent stream proxy (The publicly available explicit intent flow extraction model) is processed to extract standard operating procedures: Meanwhile, the operation command qⱼ is transmitted through the embedded model. Encoding as latent representation ,Right now: Characterization It will be stored in the retrieval database of the standard operating procedure for users. Retrieved and used during the deployment phase.

[0039] The results of extracting explicit intent streams include: from a user and the one-to-one corresponding standard operation flow. For the convenience of understanding, examples one to three are provided to illustrate the operation instruction and the standard operation flow corresponding to the operation instruction: Example one: the operation instruction is "send a new year's greetings to my leader", and the corresponding standard operation flow corresponding to the operation instruction includes "open the mobile phone, find and click the WeChat application; enter the WeChat interface, click the search box; input "leader" in the search box; select "leader" from the contact persons; input the new year's greetings content "wish the leader a happy new year, and may the team achieve better results!" in the chat box; click the send button".

[0040] Example two: the operation instruction is "tell my leader that I will attend the meeting at nine o'clock tonight", and the corresponding standard operation flow corresponding to the operation instruction includes "open the WeChat application; click the search box; input "leader" and search; select the contact person "leader"; input "received, will attend the meeting at nine o'clock tonight" in the input box; click the send button to send the message".

[0041] Example three: the operation instruction is "tell my wife that I will not go home for dinner tonight", and the corresponding standard operation flow corresponding to the operation instruction includes "open the mobile phone; click the WeChat application; click the search icon in the upper right corner; input "wife" to search; click the contact person "wife" in the search results; input "work overtime tonight, don't go back to eat, love you" in the chat box; click the send button".

[0042] In some embodiments, an implicit intent flow can be extracted from the operation screenshot sequence, the implicit intent flow including the preference information of the user; and a user habit knowledge base is constructed according to the implicit intent flow. Exemplarily, the binary tuple is processed by the implicit intent flow agent , which updates the habit knowledge base incrementally: . Wherein, the implicit intent flow agent can learn the potential behavior patterns from the interaction sequence.

[0043] The extraction result of the implicit intent flow includes: from a certain user preferences, field behavior preferences, software behavior preferences, and the like. For example, the user's software usage preferences can include: using A application to order food, using B application to navigate travel, using C application to chat, using D application to watch videos, using E application to shop, using F application to search, and using G application to listen to songs. For example, the user's field behavior preferences can include: the user likes to order a latte with fresh coconut, standard strength, normal ice, often chooses spicy hot pot, prioritizes speed, searches for nearby roast duck restaurants, and orders delivery for tomorrow morning at 9 am, etc.; the user likes to search for nearby destinations based on location.

[0044] As Figure 3 As shown in the upper half, the dual-channel processing framework of the intention flow extraction stage traverses all operation instructions in the operation instruction set until all human demonstration data are processed, and finally generates a complete habit knowledge base and a retrieval library of standard operation procedures for each user. .

[0045] Step S220: In response to the current user instruction, obtain a current interface screenshot of the terminal.

[0046] Step S230: Rewrite the current user instruction according to the pre-constructed retrieval library of standard operation procedures and the user habit knowledge base, to generate target operation instructions and target operation procedures, wherein the retrieval library includes a mapping relationship between the operation instructions and the standard operation procedures, and the user habit knowledge base includes the user's preference information.

[0047] Step S240: Input the target operation instructions, the target operation procedures, and the current interface screenshot into the intelligent agent, so that the intelligent agent performs the task requested by the current user instruction.

[0048] For specific descriptions of steps S220 to S240, please refer to steps S110 to S130, which will not be repeated here.

[0049] Please refer to Figure 4 , Figure 4 The application method is applied to 8 kinds of mobile phone GUI intelligent agents for experiments, and the experimental results are shown in the schematic diagram. The experimental results show that the IFRAgent is 6.79% better than the baseline method on average (relative improvement 32.06%) in the human intention alignment rate index, and is 5.30% better on average (relative improvement 26.34%) in the step completion rate index.

[0050] Please refer to Figure 5 , Figure 5 ​​The experimental results of applying various exemplary learning methods to the GUI agent are shown in the schematic diagram. The method of the present application is in a leading position in comparison with similar exemplary learning methods, and the various indicators are from the same level as similar methods to 10% higher than similar methods.

[0051] Please refer to Figures 6 to 8 , Figures 6 to 8 The schematic diagram of various indicators in the experiment of the method of the present application in different model parameter scales is shown. As shown in Figures 6 to 8 , the method of the present application can improve the indicators by 5% to 20% in experiments of different model parameter scales.

[0052] Please refer to Figure 9 , Figure 9 The schematic diagram of the experimental results of the experiment of the present application after modifying part of the data on the public mobile phone GUI agent evaluation benchmark OS-Kairos is shown. OS-Kairos is a new type of GUI agent system with adaptive interaction capability. The present application re-performs the data alignment standard test on part of the data on the public mobile phone GUI agent evaluation benchmark OS-Kairos, and still can improve various indicators relative to the baseline level.

[0053] Through a large number of experiments on different mobile agents, the present application proves that the IFRAgent improves the intention alignment rate by an average of 6.79% (relative improvement of 32.06%) and the step success rate by an average of 5.30% (relative improvement of 26.34%) compared with the baseline method.

[0054] Steps S210 to S240 have the following technical effects: (1) The intention flow recognition technology based on human demonstration action, according to the human demonstration data, constructs a retrieval library of standard operation process and a user habit knowledge base, which can provide the intelligent agent with explicit intention flow (i.e. the correspondence between operation instruction and standard operation process) and implicit intention flow (i.e. user preference information) of human operation at the same time, thereby improving the individualization degree and accuracy of rewriting user instructions.

[0055] (2) According to the pre-constructed retrieval library of standard operation process and user habit knowledge base, the current user instruction is rewritten to generate target operation instruction and target operation process, so that the target operation instruction and target operation process conform to the explicit intention (e.g. operation logic, step sequence) and implicit intention (e.g. user habit, personal preference) of the user, thereby realizing the accurate alignment of the intention of the intelligent agent and the human, enabling the intelligent agent to highly understand the real intention of different users, and improving the ability and individualization degree of the intelligent agent to simulate human operation to complete tasks.

[0056] Please refer to Figure 10 ,Figure 10 A structural block diagram of an agent-based interaction device is shown. As shown, the agent-based interaction device 100 can include an information acquisition module 110, an information rewriting module 120, and an action execution module 130. Figure 10

[0057] The information acquisition module 110 is configured to acquire a current interface screenshot of the terminal in response to a current user instruction.

[0058] The information rewriting module 120 is configured to rewrite the current user instruction according to a retrieval library of standard operation procedures and a user habit knowledge base, to generate a target operation instruction and a target operation procedure, wherein the retrieval library of standard operation procedures includes a mapping relationship between operation instructions and standard operation procedures, and the user habit knowledge base includes preference information of the user.

[0059] The action execution module 130 is configured to input the target operation instruction, the target operation procedure, and the current interface screenshot into an agent, to enable the agent to perform a task requested by the current user instruction.

[0060] In some embodiments, the information rewriting module 120 is further configured to acquire, from the retrieval library of standard operation procedures, an operation instruction most similar to the current user instruction as a most similar instruction, and acquire a standard operation procedure corresponding to the most similar instruction; and generate the target operation instruction and the target operation procedure according to the most similar instruction, the standard operation procedure corresponding to the most similar instruction, the current user instruction, and the user habit knowledge base.

[0061] In some embodiments, the information rewriting module 120 is further configured to determine a standard operation procedure corresponding to the current user instruction according to the most similar instruction, the standard operation procedure corresponding to the most similar instruction, and the current user instruction; and generate the target operation instruction and the target operation procedure according to the current user instruction, the standard operation procedure corresponding to the current user instruction, and the user habit knowledge base.

[0062] In some embodiments, the information rewriting module 120 is further configured to perform similarity matching between an operation instruction in the retrieval library of standard operation procedures and the current user instruction; and acquire an operation instruction most similar to the current user instruction as a most similar instruction when there is an operation instruction whose similarity to the current user instruction is higher than a similarity threshold.

[0063] ​In some embodiments, the agent-based interaction apparatus 100 can further comprise a database construction module. The database construction module is configured to: before obtaining the current user instruction and the current interface screenshot, construct a retrieval library of standard operation flows and a user habit knowledge base according to human demonstration data, wherein the human demonstration data comprises operation instructions and a sequence of operation screenshots made by a user for the operation instructions.

[0064] In some embodiments, the database construction module is further configured to: extract an explicit intent flow from the sequence of operation screenshots, the explicit intent flow comprising operation instructions and a standard operation flow corresponding to the operation instructions; and construct the retrieval library of standard operation flows according to the explicit intent flow.

[0065] In some embodiments, the database construction module is further configured to: extract an implicit intent flow from the sequence of operation screenshots, the implicit intent flow comprising preference information of the user; and construct the user habit knowledge base according to the implicit intent flow.

[0066] It can be clearly understood by those skilled in the art that the agent-based interaction apparatus 100 provided by the embodiments of the present application can implement the agent-based interaction method provided by the embodiments of the present application. The specific working process of the above apparatus and modules can refer to the process corresponding to the agent-based interaction method in the embodiments of the present application, which will not be repeated here.

[0067] In the embodiments provided in the present application, the coupling, direct coupling or communication connection between the modules displayed or discussed can be indirect coupling or communication coupling through some interfaces, devices or modules, and can be electrical, mechanical or other forms, which are not limited in the embodiments of the present application.

[0068] In addition, each functional module in the embodiments of the present application can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module, which is not limited in the embodiments of the present application.

[0069] Please refer to Figure 11 , Figure 11 is a structural block diagram of an electronic device provided by an embodiment of the present application. As Figure 11 shown, the electronic device 200 can include a memory 210 and a processor 220. The memory 210 stores an application program, which is configured to make the processor 220 execute the above-mentioned agent-based interaction method provided by the embodiments of the present application when called by the processor 220.

[0070] The processor 220 can include one or more processing cores. The processor 220 connects various parts within the entire electronic device 200 with various interfaces and lines, for running or executing instructions, programs, code sets or instruction sets stored in the memory 210, calling running or executing data stored in the memory 210, performing various functions and processing data of the electronic device 200.

[0071] Optionally, the processor 220 can be implemented in at least one of a hardware form of a Digital Signal Processing (DSP), a Field-Programmable Gate Array (FPGA), and a Programmable Logic Array (PLA). The processor 220 can be integrated with a combination of one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. Among them, the CPU is mainly used to process operating systems, user interfaces, and application programs, etc. The GPU is used to be responsible for rendering and drawing display content. The modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated in the processor 220, but be realized by a separate communication chip.

[0072] The memory 210 can include a Random Access Memory (RAM) and can also include a Read-Only Memory (ROM). The memory 210 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 210 can include a program storage area and a data storage area. Among them, the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the above-mentioned various method embodiments, etc. The data storage area can store data created by the electronic device 200 in use, etc.

[0073] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium has program codes stored thereon, and the program codes are used to cause the processor to execute the above-mentioned interaction method based on an agent provided by the embodiments of the present application when the processor calls the program codes.

[0074] The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer readable storage medium can include, but are not limited to, an electrical connection based on one or more of a semiconductor based or other integrated circuit (IC) (such as for example, but not limited to, a system-on-chip (SoC), an application-specific IC (ASIC), a microcontroller, etc.), a programmable logic device (PLD), or a programmable logic array (PLA), a hard disk, a floppy disk, a CD-ROM, a DVD, a Blu-ray disk, a flash memory, an EEPROM, an EPROM, a RAM, a ROM, or any suitable combination of the foregoing. A computer readable storage medium can optionally include a non-transitory computer readable storage medium (Non-TCRSM).

[0075] The computer readable storage medium has storage space for program codes to perform any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program codes can be compressed in an appropriate form.

[0076] In summary, the embodiments of the present application provide an interactive method and device based on an agent, an electronic device, and a storage medium. The current user instruction is rewritten according to a retrieval library of a standard operation process and a user habit knowledge base, and a target operation instruction and a target operation process are generated. The target operation instruction and the target operation process conform to the explicit intention (for example, operation logic and step sequence) and the implicit intention (for example, user habit and personal preference) of the user, so that the accurate alignment of the agent and the human intention is realized, the real intention of different users can be highly understood by the agent, and the ability of the agent to complete a task by simulating human operation and the individualization degree are improved.

[0077] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to part of the technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An agent-based interaction method, characterized in that, include: Responding to the current user command, obtain a screenshot of the current interface of the terminal; Based on a pre-built standard operation procedure retrieval library and a user habit knowledge base, the current user instruction is rewritten to generate a target operation instruction and a target operation procedure. The retrieval library includes a mapping relationship between operation instructions and standard operation procedures, and the user habit knowledge base includes user preference information. The target operation instruction, the target operation process, and the current interface screenshot are input into the intelligent agent so that the intelligent agent can execute the task requested by the current user instruction.

2. The method according to claim 1, characterized in that, The step of rewriting the current user command based on a pre-built standard operation procedure retrieval library and user habit knowledge base to generate target operation commands and target operation procedures includes: From the standard operation procedure retrieval library, obtain the operation instruction most similar to the current user instruction as the most similar instruction, and obtain the standard operation procedure corresponding to the most similar instruction; Based on the most similar instruction, the standard operation procedure corresponding to the most similar instruction, the current user instruction, and the user habit knowledge base, a target operation instruction and a target operation procedure are generated.

3. The method according to claim 2, characterized in that, The step of generating target operation instructions and target operation procedures based on the most similar instruction, the standard operation procedure corresponding to the most similar instruction, the current user instruction, and the user habit knowledge base includes: Based on the most similar instruction, the standard operation procedure corresponding to the most similar instruction, and the current user instruction, determine the standard operation procedure corresponding to the current user instruction; Based on the current user instruction, the standard operation procedure corresponding to the current user instruction, and the user habit knowledge base, generate the target operation instruction and the target operation procedure.

4. The method according to claim 2, characterized in that, The step of retrieving the operation instruction most similar to the current user instruction from the search database as the most similar instruction includes: The operation instructions in the standard operation procedure search library are matched with the current user instructions based on similarity. If an operation instruction has a similarity higher than the current user instruction, the operation instruction with the highest similarity to the current user instruction is selected as the most similar instruction.

5. The method according to any one of claims 1-4, characterized in that, Before obtaining the current user command and the current interface screenshot, the method further includes: Based on human demonstration data, a retrieval library for standard operating procedures and a user habit knowledge base are constructed. The human demonstration data includes operating instructions and a sequence of screenshots of user actions in response to the operating instructions.

6. The method according to claim 5, characterized in that, Based on human demonstration data, a retrieval library for standard operating procedures was constructed, including: From the sequence of operation screenshots, an explicit intent stream is extracted, which includes operation instructions and standard operation procedures corresponding to the operation instructions. Based on the explicit intent flow, a retrieval library for standard operating procedures is constructed.

7. The method according to claim 5, characterized in that, Based on human demonstration data, a user habit knowledge base is constructed, including: From the sequence of operation screenshots, an implicit intent stream is extracted, which includes the user's preference information; Based on the implicit intent flow, a user habit knowledge base is constructed.

8. An interactive device based on an intelligent agent, characterized in that, include: The information acquisition module is used to respond to the current user's command and acquire a screenshot of the current interface of the terminal; The information rewriting module is used to rewrite the current user instruction based on a pre-built standard operation procedure retrieval library and a user habit knowledge base, to generate a target operation instruction and a target operation procedure. The retrieval library includes a mapping relationship between operation instructions and standard operation procedures, and the user habit knowledge base includes user preference information. The action execution module is used to input the target operation instruction, the target operation process, and the current interface screenshot into the intelligent agent, so that the intelligent agent can execute the task requested by the current user instruction.

9. An electronic device, characterized in that, include: A memory and a processor, wherein an application program is stored in the memory, the application program being invoked by the processor to cause the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that, when invoked by a processor, causes the processor to perform the method as described in any one of claims 1-7.

Images