Software interaction method, device and equipment based on AI, RPA and AI Agent

Abstract

The invention provides a software interaction method, device and equipment based on AI, RPA and AI Agent, and relates to the field of AI, RPA and AI Agent. The method comprises the steps of obtaining dialogue information from a dialogue application; identifying the dialogue information to obtain to-be-interacted software and interaction operation; calling a standard interaction component matched with the software and the interaction operation from a standard interaction component library embedded in the large language model; displaying the standard interaction component in the current dialogue; and executing interaction operation on the software based on the interaction information triggered for the standard interaction component. Therefore, the standard interaction component is called and displayed in the dialogue application based on the dialogue information in the natural language form input by the user, and the existing software is quickly driven and remodeled based on the interaction information of the user and the standard interaction component, so that the convenience and the interaction efficiency of software interaction can be improved. Besides, software interaction can be realized based on AI, RPA and AI Agent technologies, and the accuracy of software interaction is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence (AI), robotic process automation (RPA) and artificial intelligence agent (AI agent), and in particular to a software interaction method, device and equipment based on AI, RPA and AI agent. Background Art

[0002] Robotic Process Automation (RPA) uses specific "robot software" to simulate human operations on computers and automatically execute process tasks according to rules.

[0003] Artificial Intelligence (AI) is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0004] The Artificial Intelligence Agent (AI Agent) of the digital labor platform can perceive the environment, make decisions and perform actions. The difference from traditional artificial intelligence is that it has the ability to think and act independently and can call tools to complete given goals. AI Agent is based on Large Language Models (LLM) as its core computing engine, which enables it to have conversations, perform tasks, reason and show a certain degree of autonomy. It has the ability to understand, perceive, plan, remember and use tools independently, and can automatically perform complex tasks. Specifically, AI Agent, an intelligent assistant composed of multiple AI capabilities driven by LLM, can interact with employees in natural language, understand employees' instructions and needs, and provide feedback and responses; it can master domain knowledge related to the business to complete complex professional work; it can decompose complex tasks into several executable tasks and use data and tools to complete tasks; it can also collaborate with employees, and AI Agents can also collaborate to complete complex tasks, so as to enable digital employees to achieve a leap from automation to intelligence, help employees complete their work more efficiently, and fully realize human-machine collaboration.

[0005] In related technologies, low-code platforms are used to transform existing software systems into new intelligent interactive software systems. Among them, low-code platforms attempt to achieve software interaction through intuitive and visual interface components and workflow building methods, reduce software development costs, and use drag-and-drop methods to build software interfaces, and implement complex functions related to software interfaces through visual workflow orchestration.

[0006] However, the low-code platform is a programming paradigm oriented towards the traditional graphical user interface (GUI, also known as graphical user interface, which refers to a computer operation user interface displayed in graphical form). The interface logic is highly complex and the interaction method is relatively inconvenient. Summary of the invention

[0007] The embodiment of the present application provides a software interaction method, device and equipment based on AI, RPA and AI Agent to solve one of the technical problems existing in the related technology. The technical solution is as follows:

[0008] In a first aspect, an embodiment of the present application provides a software interaction method based on AI, RPA, and AI Agent, including:

[0009] Acquire the dialogue information from the dialogue application, and identify the dialogue information to obtain the first software to be interacted and the first interactive operation;

[0010] Calling a first standard interaction component adapted to the first software and the first interaction operation from a standard interaction component library embedded in the large language model;

[0011] Displaying the first standard interaction component in the current conversation of the conversation application;

[0012] Based on the first interaction information triggered by the first standard interaction component, the first interaction operation is performed on the first software.

[0013] In a second aspect, an embodiment of the present application provides a software interaction device based on AI, RPA and AI Agent, including:

[0014] An identification module, used to obtain dialogue information from the dialogue application and identify the dialogue information to obtain the first software to be interacted and the first interaction operation;

[0015] A calling module, configured to call a first standard interaction component adapted to the first software and the first interaction operation from a standard interaction component library embedded in a large language model;

[0016] A display module, used for displaying the first standard interaction component in the current dialog of the dialog application;

[0017] An execution module is used to execute the first interaction operation on the first software based on the first interaction information triggered by the first standard interaction component.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a memory and a processor. The memory and the processor communicate with each other through an internal connection path, the memory is used to store instructions, the processor is used to execute the instructions stored in the memory, and when the processor executes the instructions stored in the memory, the processor executes the method in any one of the above-mentioned embodiments.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer, the method in any one of the above-mentioned embodiments is executed.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method in any one of the above-mentioned aspects when executed by a processor.

[0021] The advantages or beneficial effects of the above technical solution include at least:

[0022] Based on the dialogue information in the form of natural language input by the user, standard interaction components are called and displayed in the dialogue application, and based on the interaction information between the user and the standard interaction components, the existing software is quickly driven and reshaped, that is, the interaction operations required by the user are performed on the existing software. Since the input efficiency of dialogue information in the form of natural language is high, and the interaction method of standard interaction components is simpler and more intuitive, the convenience and efficiency of software interaction can be improved, and the user experience can be improved. In addition, software interaction can be realized based on AI, RPA and AI Agent technologies to improve the accuracy of software interaction.

[0023] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0025] Figure 1 It is a schematic diagram of a GUI interaction method in the related art;

[0026] Figure 2 It is a schematic diagram of the interaction method of the Copilot mode in the related art;

[0027] Figure 3 It is a flowchart of a software interaction method based on AI, RPA and AI Agent provided by an embodiment of the present application;

[0028] Figure 4 is a flowchart of a software interaction method based on AI, RPA and AI Agent provided by another embodiment of the present application;

[0029] Figure 5 is a flowchart of a software interaction method based on AI, RPA and AI Agent provided by another embodiment of the present application;

[0030] Figure 6 is a flowchart of a software interaction method based on AI, RPA and AI Agent provided by another embodiment of the present application;

[0031] Figure 7 This is a schematic diagram of a framework for implementing a fully intelligent fusion interaction of a conversational GUI provided by an embodiment of the present application;

[0032] Figure 8 It is a structural diagram of a software interaction device based on AI, RPA and AI Agent provided by one embodiment of the present application;

[0033] Fig. 9 A structural block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0035] In the related technologies, the following two methods are mainly used to realize the transformation of the existing software system to the new intelligent interactive software system:

[0036] The first one is the low-code platform.

[0037] Among them, low-code platforms attempt to implement software through intuitive and visual interface components and workflow building methods to reduce the cost of software development. In addition, low-code platforms generally use a drag-and-drop method to build software interfaces. In order to implement complex functional logic, complex interface-related functions can be implemented through visual workflow arrangement.

[0038] The second is to automate software interactions through the Copilot (auxiliary assistant or collaborative assistant) mode.

[0039] Among them, the Copilot mode hopes to drive the existing software interface through intelligent dialogue, and the dialogue assistant floats above the existing software to intelligently drive and reconstruct the existing software functions.

[0040] As an example, the traditional graphical user interface (GUI) interaction method can be as follows: Figure 1 As shown, the Copilot mode interaction can be as follows Figure 2 shown.

[0041] However, the first method mentioned above has at least the following problems:

[0042] First, it is still a programming paradigm for traditional GUIs. It is not a thorough interaction method based on natural language dialogue. It cannot fully utilize natural language dialogue based on a large language model to reduce the complexity of interface logic and improve adaptability.

[0043] The second item simplifies traditional interface and logic code programming through visual drag-and-drop interfaces and workflows, but does not consider the problem of how to quickly integrate and reconstruct traditional software, especially cross-device software without an application programming interface (API).

[0044] The second method mentioned above has at least the following problems:

[0045] First, the GUI interaction of the backend software is not completely integrated with the dialogue. The dialogue robot is suspended or embedded in the traditional software interface. When interacting, the user needs to switch between the dialogue and the backend software GUI, and the interaction is not very smooth.

[0046] Second, there is no implementation method to quickly integrate traditional software into the dialogue and GUI fusion interaction framework for reconstruction, especially for those cross-device software that do not have API interfaces.

[0047] In order to solve at least one of the above-mentioned problems, the embodiments of the present application propose a software interaction method, device and equipment based on artificial intelligence (AI), robotic process automation (RPA) and digital labor platform intelligent assistant (AIAgent). These and other aspects of the embodiments of the present application will be clear with reference to the following description and drawings. In these descriptions and drawings, some specific implementations in the embodiments of the present application are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present application, but it should be understood that the scope of the embodiments of the present application is not limited thereto. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0048] Before describing the embodiments of the present application in detail, in order to facilitate understanding, common technical terms are first introduced:

[0049] In the description of the present application, the term "plurality" refers to two or more.

[0050] In the description of this application, the term "large language model" refers to a type of natural language processing model based on deep learning. Its main characteristics are that it has huge model parameters and complex neural network structure, and has strong language understanding, context perception and language generation capabilities. It can automatically learn useful feature representations from input data and generate relevant text.

[0051] In the description of this application, the term "dialogue application" refers to a dialogue application, which is an application program (Application, APP for short) that allows users to interact or dialogue with the application through natural language.

[0052] In the description of this application, the term "first software" refers to the software that the user actually wants to interact with, which is determined based on the user's conversation intention. Correspondingly, the term "second software" refers to the software that the user actually wants to interact with, which is re-determined based on the user's conversation intention.

[0053] In the description of this application, the term "first interactive operation" refers to an operation that a user wants to interact with a first software, including but not limited to: a login operation, a query operation, a search operation, etc. Correspondingly, the term "second interactive operation" refers to an operation that a user wants to interact with a second software.

[0054] In the description of this application, the term "standard interaction component library" refers to a component library provided with multiple standard interaction components (such as standard GUI components, standard user interface (UI) components), wherein the standard interaction components can be adapted to various types of software. For example, the standard interaction components may include a standard user name input component (for inputting a user name in login information), a standard password input component (for inputting a password in login information), a standard verification code input component (a user inputs a verification code in identity authentication information), etc., wherein the standard user name input component and the standard password input component can be adapted to login scenarios of all software, and the standard verification code input component can be adapted to identity authentication scenarios under login scenarios of all software.

[0055] In the description of this application, the term "interaction interface" refers to an interface for operating software, or an interface for interacting with software. Exemplarily, the interaction interface may include an API interface of the software, or an RPA automation agent interface associated with the software, wherein the RPA automation agent interface refers to an agent interface for operating a software UI that does not include an API interface.

[0056] Among them, the RPA automation agent interface can be an atomic interface, or it can be a combined interface assembled from multiple atomic interfaces, which is not limited in the embodiments of the present application.

[0057] In the description of this application, the term "feedback information" refers to information manually input by a user when the interaction result of the software does not meet the user's expectations. The feedback information is used to indicate the user's actual conversation intention.

[0058] These and other aspects of the embodiments of the present application will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations in the embodiments of the present application are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present application, but it should be understood that the scope of the embodiments of the present application is not limited thereto. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0059] The following describes the software interaction method, device and equipment based on AI, RPA and AI Agent according to the embodiments of the present application in conjunction with the accompanying drawings.

[0060] Figure 3 This is a flowchart of a software interaction method based on AI, RPA and AI Agent provided by an embodiment of the present application.

[0061] In a possible implementation method provided in an embodiment of the present application, the present application takes the software interaction method based on AI, RPA and AI Agent as an example, and the software interaction device based on AI, RPA and AI Agent is configured in the software interaction device. The software interaction device can be applied to any electronic device with computing capabilities.

[0062] The electronic device may be a personal computer, a mobile terminal, a service end (or server), etc. The mobile terminal may be, for example, a mobile phone, a tablet computer, a personal digital assistant, or other hardware device with various operating systems.

[0063] In another possible implementation of the embodiment of the present application, the software interaction method based on AI, RPA and AI Agent can be applied to AI Agent, wherein the AI ​​Agent can run in any electronic device with computing capabilities.

[0064] In another possible implementation of the embodiment of the present application, the software interaction method based on AI, RPA and AI Agent can be applied to an intelligent automation platform; wherein the intelligent automation platform can realize seamless integration of multiple capabilities such as RPA, intelligent document processing (IDP), conversational AI (CoAI), and process mining (Process Mining), and has five major functions of "business understanding", "process creation", "run anywhere", "centralized control", and "human-machine collaboration", which can realize end-to-end intelligent automation of business processes for enterprises, replace manual operations, further improve business efficiency, and accelerate digital transformation.

[0065] Among them, intelligent document processing (IDP) is one of the core capabilities of the intelligent automation platform. Intelligent document processing (IDP) is a new generation of automation technology based on AI technologies such as optical character recognition (OCR), computer vision (CV), natural language processing (NLP), and knowledge graph (KG). It can identify, classify, extract elements, verify, compare, and correct errors of various documents, helping enterprises realize the intelligence and automation of document processing.

[0066] like Figure 3 As shown, the software interaction method based on AI, RPA and AI Agent may include the following steps S301 to S304:

[0067] Step S301: Acquire dialogue information from a dialogue application, and identify the dialogue information to obtain a first software to be interacted with and a first interactive operation.

[0068] In an embodiment of the present application, conversation information input by a user may be obtained from a conversation application, wherein the input method of the conversation information includes, but is not limited to, touch input (such as sliding, clicking, etc.), keyboard input, voice input, etc.

[0069] In an embodiment of the present application, the above-mentioned dialogue information can also be identified to obtain the first software that the user actually wants to interact with and the specific operation of interacting with the first software (referred to as the first interaction operation in this application).

[0070] For example, assuming that the dialogue information is "check my order list today in shopping software A", the first software that the user wants to interact with is "shopping software A", and the first interactive operation includes "login operation" and "query today's order list operation".

[0071] As an example, the above-mentioned conversation information can be subjected to intent recognition to obtain the user's actual conversation intent. For example, the conversation intent to which the conversation information belongs can be identified based on the intent recognition algorithm or large language model in deep learning technology. The conversation intent can be used to indicate the first software that the user wants to interact with and the specific operation (i.e., the first interaction operation) for interacting with the first software.

[0072] Step S302: calling a first standard interaction component adapted to the first software and the first interaction operation from a standard interaction component library embedded in the large language model.

[0073] The number of the first standard interaction components may be one or more, and the embodiment of the present application does not limit this. It should be noted that, for different dialogue intentions, even if the interaction operations indicated by these dialogue intentions are the same, the standard interaction components adapted to the software and interaction operations indicated by these dialogue intentions may not be completely the same.

[0074] For example, suppose the dialogue Figure 1 The software to be interacted is software 1, and the dialogue Figure 2 The software to be interacted with is software 2, and the dialogue Figure 1 and dialogue Figure 2The indicated interactive operations are all login operations. Since the login scenario of software 1 does not require the user to enter a verification code, while the login scenario of software 2 requires the user to enter a verification code, the standard interactive components adapted to software 1 and the login operation may include a standard user name input component and a standard password input component, and the standard interactive components adapted to software 2 and the login operation may include a standard user name input component, a standard password input component, and a standard verification code input component.

[0075] In an embodiment of the present application, a standard interaction component (referred to as the first standard interaction component in the present application) adapted to the first software and the first interactive operation can be determined and called from a plurality of standard interaction components in a standard interaction component library embedded in a large language model.

[0076] Step S303: Display the first standard interaction component in the current conversation of the conversation application.

[0077] In the embodiment of the present application, the called first standard interaction component can be displayed in the current conversation of the conversation application, that is, the first standard interaction component is called out in the current conversation of the conversation application.

[0078] Step S304: Based on the first interaction information triggered by the first standard interaction component, a first interaction operation is performed on the first software.

[0079] In an embodiment of the present application, in response to an interactive operation triggered by a user on a first standard interactive component displayed in a current conversation, first interaction information may be acquired, and based on the first interaction information, a first interactive operation may be performed on the first software.

[0080] Still taking the example in step S301 as an example, the first standard interaction component may include a standard user name input component and a standard password input component. The first interaction information triggered by the user for the standard user name input component may include the user's user name in "A shopping software", and the first interaction information triggered by the user for the standard password input component may include the login password corresponding to the user name. In the present application, based on the first interaction information including the user name and the login password, a login operation may be performed on "A shopping software" first. If the login is successful, based on the user name in the first interaction information, the user's order list today in "A shopping software" is queried, and the order list is provided to the user, so that the user can intuitively know the query results.

[0081] The software interaction method based on AI, RPA and AI Agent in the embodiment of the present application can realize dialogue information in the form of natural language input by the user, call and display standard interaction components in the dialogue application, and quickly drive and reshape the existing software based on the interaction information between the user and the standard interaction components, that is, perform the interaction operations required by the user on the existing software. Since the input efficiency of dialogue information in the form of natural language is high, and the interaction method of the standard interaction components is simpler and more intuitive, it can improve the convenience and interaction efficiency of software interaction and improve the user experience. In addition, software interaction can be realized based on AI, RPA and AI Agent technology to improve the accuracy of software interaction.

[0082] In order to clearly illustrate how in any embodiment of the present application, a first standard interaction component adapted to the first software and the first interaction operation is called from a standard interaction component library embedded in a large language model, the present application also proposes a software interaction method based on AI, RPA and AI Agent.

[0083] Figure 4 It is a flowchart of a software interaction method based on AI, RPA and AI Agent provided in another embodiment of the present application.

[0084] It should be noted that the software interaction method based on AI, RPA and AI Agent can be executed alone, or it can be executed in combination with any embodiment in the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the relevant technology, and the embodiments of the present application are not limited to this.

[0085] like Figure 4 As shown, the software interaction method based on AI, RPA and AI Agent may include the following steps S401 to S408:

[0086] Step S401: Acquire dialogue information from a dialogue application, and identify the dialogue information to obtain a first software to be interacted with and a first interactive operation.

[0087] Step S402: Retrieve candidate software with semantic similarity to the dialogue information from multiple interactive software.

[0088] The number of candidate software may be at least one.

[0089] In the embodiment of the present application, a retrieval-augmented generation (RAG) technology may be used to retrieve candidate software with semantic similarity to the dialog information from multiple interactive software.

[0090] Step S403: Retrieve a candidate interface with semantic similarity to the dialog information from a plurality of interactive interfaces associated with the candidate software.

[0091] Wherein, in the case where a candidate software includes an API interface, the interactive interface includes the API interface of the candidate software; in the case where a candidate software does not include an API interface, the interactive interface includes an RPA automation agent interface associated with the candidate software, wherein the RPA automation agent interface is used to operate the user interface of the candidate software, that is, the RPA automation agent interface refers to an agent interface that simulates the operation of the candidate software UI. Therefore, even if the software does not include an API interface, users can interact with the software based on the RPA automation agent interface to meet the personalized software interaction needs of different users.

[0092] That is, in an embodiment of the present application, when the candidate software includes an API interface, the API interface can be used as an interaction interface, and when the candidate software does not include an API interface, the RPA automation agent interface associated with the candidate software can be used as an interaction interface. Therefore, in the present application, RAG technology can be used to retrieve candidate interfaces that are semantically similar to the dialogue information from multiple interaction interfaces associated with the candidate software.

[0093] Step S404: calling the large language model to determine the first software from the candidate software according to the dialogue information, and determining the target interface from the candidate interfaces.

[0094] In an embodiment of the present application, a large language model may be called to determine the first software that the user wants to interact with from the candidate software based on the dialogue information, and to determine the target interface to be called from the candidate interfaces.

[0095] In any embodiment of the present application, the first software and the target interface can be determined by following steps A to C:

[0096] Step A: Obtain interface information and interface parameters of the candidate interface.

[0097] The interface information includes, but is not limited to, interface name, interface description, request method, etc.

[0098] The interface parameters include but are not limited to request parameters, response parameters, etc.

[0099] Step B: Generate prompt information based on the dialogue information, candidate software, interface information and interface parameters; wherein the prompt information is used to indicate task information to be executed by the large language model.

[0100] As an example, the dialogue information, the software information of the candidate software (such as software identification, software function description, etc.), the interface information and interface parameters of multiple candidate interfaces can be assembled to obtain prompt information (Prompt), wherein the Prompt is used to prompt the task information to be executed by the large language model (such as instructing the language model to execute the call mapping process generation task).

[0101] Step C: Process the prompt information using a large language model to obtain the first software and the target interface.

[0102] As an example, a large language model can be called to process the Prompt to obtain a call mapping process output by the large language model; wherein the call mapping process is at least used to indicate the mapping relationship between the first software to be interacted, the target interface in the candidate interface, the interactive operation associated with the target interface, and the interface parameters of the target interface.

[0103] Therefore, planning the call mapping process based on the powerful natural semantic understanding, generation and planning capabilities of the large language model can improve the planning accuracy of the call mapping process and thus improve the accuracy of subsequent software interactions.

[0104] It should be noted that when a large language model is used to orchestrate the call mapping process, if the candidate interface is an atomic interface, the number of orchestrations of the large language model will increase, and the error rate of the orchestration results output by the large language model will increase. In response to this situation, in any embodiment of the present application, multiple atomic interfaces can be combined to obtain a candidate interface (such as an RPA automation agent interface) to improve the orchestration accuracy of the large language model.

[0105] In the embodiment of the present application, the first software to be interacted and the target interface can be determined from the call mapping process.

[0106] Step S405: taking the interactive operation associated with the target interface as the first interactive operation.

[0107] As an example, the interactive operation associated with the target interface indicated by the call mapping process may be taken as the first interactive operation.

[0108] Step S406: calling a first standard interaction component adapted to the first software and the first interaction operation from a standard interaction component library embedded in the large language model.

[0109] In any embodiment of the present application, the prompt information may also include component information of multiple standard interaction components in the standard interaction component library, wherein the component information of the standard interaction component includes but is not limited to: component identification (such as component name, component ID, etc.), function description information, input format, output format, etc.; that is, in the present application, prompt information can be generated according to dialogue information, software information of candidate software, interface information and interface parameters of candidate interfaces, and component information of multiple standard interaction components, and the prompt information is processed using a large language model to obtain a call mapping process. At this time, the call mapping process is specifically used to indicate the mapping relationship between the first software to be interacted, the target interface in the candidate interface, the interaction operation associated with the target interface, the interface parameters of the target interface, and the standard interaction components to be interacted in the standard interaction components.

[0110] In the present application, the standard interaction component to be interacted indicated by the calling mapping process can be used as the first standard interaction component, and the first standard interaction component can be called based on the interface parameters of the target interface indicated by the calling mapping process.

[0111] Step S407: display the first standard interaction component in the current conversation of the conversation application.

[0112] Step S408: performing a first interactive operation on the first software based on the first interactive information triggered by the first standard interactive component.

[0113] The explanation of steps S407 to S408 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0114] The software interaction method based on AI, RPA and AI Agent in the embodiment of the present application adopts RAG technology to preliminarily screen candidate software and candidate interfaces with similar semantics to dialogue information, and calls the large language model to determine the first software to be interacted from the candidate software according to the dialogue information, and determines the target interface to be called from the candidate interface, and takes the interaction operation associated with the target interface as the first interaction operation, which can not only reduce the recognition range of the large language model, but also improve the recognition accuracy of the large language model.

[0115] In order to clearly illustrate how the first interactive operation is performed on the first software in any embodiment of the present application, the present application also proposes a software interaction method based on AI, RPA and AI Agent.

[0116] Figure 5 It is a flowchart of a software interaction method based on AI, RPA and AI Agent provided in another embodiment of the present application.

[0117] It should be noted that the software interaction method based on AI, RPA and AI Agent can be executed alone, or it can be executed in combination with any embodiment in the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the relevant technology, and the embodiments of the present application are not limited to this.

[0118] like Figure 5 As shown, the software interaction method based on AI, RPA and AI Agent may include the following steps S501 to S506:

[0119] Step S501: Acquire dialogue information from a dialogue application, and identify the dialogue information to obtain a first software to be interacted with and a plurality of first interactive operations.

[0120] For the explanation of step S501, please refer to the relevant description in any embodiment of the present application, and will not be repeated here.

[0121] In an embodiment of the present application, the number of first interaction operations may be multiple. For example, taking the dialogue information as "Check my order list today in shopping software A", the first software to be interacted with indicated by the dialogue intention is "A shopping software", and the first interaction operation includes two operations: "login operation" and "query today's order list operation". For another example, taking the dialogue information as "Check my unread messages in social software B", the first software to be interacted with indicated by the dialogue intention is "B social software", and the first interaction operation includes two operations: "login operation" and "query unread messages operation".

[0122] Step S502: calling a first standard interaction component adapted to the first software and the plurality of first interaction operations from a standard interaction component library embedded in the large language model.

[0123] For the explanation of step S502, please refer to the relevant description in any embodiment of the present application, and will not be repeated here.

[0124] The first standard interaction component may include an input interaction component and an output interaction component, wherein the input interaction component is used for a user to input interaction information, and the output interaction component is used to output or display the interaction result of the first software.

[0125] Step S503: Display the first standard interaction component in the current conversation of the conversation application.

[0126] For the explanation of step S503, please refer to the relevant description in any embodiment of the present application, and will not be repeated here.

[0127] Step S504: Acquire first interaction information triggered by the input interaction component displayed in the current dialog.

[0128] In the embodiment of the present application, the user can operate the first software by interacting with the input interaction component displayed in the current conversation. Accordingly, the interaction information triggered by the user for the input interaction component (referred to as the first interaction information in the present application) can be obtained.

[0129] Step S505: The first interaction information is transmitted to the target interface, so as to perform a plurality of first interaction operations on the first software through the target interface to obtain an interaction result.

[0130] In the embodiment of the present application, the first interaction information may be input into the target interface so as to perform a plurality of first interaction operations on the first software through the target interface to obtain an interaction result.

[0131] Still taking the example in step S501 as an example, assuming that the first interaction operation includes two operations, "login operation" and "query unread message operation", the interaction result includes all the unread messages of the user currently in B social software.

[0132] In any embodiment of the present application, when there are multiple target interfaces, the calling order between the multiple target interfaces can also be obtained. Exemplarily, the calling mapping process output by the large language model can also include: the calling order between the multiple target interfaces. In the present disclosure, the calling order between the multiple target interfaces can be queried from the calling mapping process. For step S504, the first interaction information can be transmitted to the multiple target interfaces in sequence based on the calling order between the multiple target interfaces, and the execution result of the last target interface can be used as the interaction result.

[0133] Wherein, for the first target interface, the first target interface may execute a first interactive operation having a mapping relationship with the first target interface on the first software based on the first interactive information to obtain an execution result of the first target interface.

[0134] For a non-first target interface (such as the i-th target interface, where i is a positive integer greater than 1), the i-th target interface can execute a first interactive operation that has a mapping relationship with the i-th target interface on the first software based on the execution results of the previous i-1 target interfaces, the historical conversation information within the conversation application, and the first interaction information, to obtain the execution result of the i-th target interface.

[0135] Exemplarily, the large language model may summarize the input parameters of the i-th target interface according to the execution results of the previous i-1 target interfaces, the historical dialogue information, and the first interaction information.

[0136] Therefore, according to the calling order between the multiple target interfaces, the multiple target interfaces are called in sequence to interact with the first software, which can improve the effectiveness and accuracy of the software interaction.

[0137] Step S506: Display the interaction result through the output interaction component displayed in the current conversation.

[0138] In the embodiment of the present application, the above interaction result can be displayed through the output interaction component displayed in the current conversation.

[0139] The software interaction method based on AI, RPA and AI Agent in the embodiment of the present application can perform the first interaction operation on the first software by calling the target interface (i.e., RPA automation agent interface) associated with the first software, even if the first software does not contain an API interface, and can meet the personalized software interaction needs of different users. In addition, based on a visual method, the software interaction results are displayed in a timely manner, so that users can intuitively know the software interaction results and meet the software interaction needs of users.

[0140] In order to clearly illustrate the above embodiments of the present application, the present application also proposes a software interaction method based on AI, RPA and AI Agent.

[0141] Figure 6 It is a flowchart of a software interaction method based on AI, RPA and AI Agent provided in another embodiment of the present application.

[0142] It should be noted that the software interaction method based on AI, RPA and AI Agent can be executed alone, or it can be executed in combination with any embodiment in the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the relevant technology, and the embodiments of the present application are not limited to this.

[0143] like Figure 6 As shown, the software interaction method based on AI, RPA and AI Agent may include the following steps S601 to S609:

[0144] Step S601: Acquire dialogue information from a dialogue application, and identify the dialogue information to obtain a first software to be interacted with and a first interactive operation.

[0145] Step S602: calling a first standard interaction component adapted to the first software and the first interaction operation from a standard interaction component library embedded in the large language model.

[0146] Step S603: Display the first standard interaction component in the current conversation of the conversation application.

[0147] Step S604: Based on the first interaction information triggered by the first standard interaction component, a first interaction operation is performed on the first software.

[0148] The explanation of steps S601 to S604 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0149] Step S605: receiving feedback information input in response to the interaction result not meeting expectations.

[0150] In an embodiment of the present application, when the interaction result of the first software does not meet the user's expectations, the user can input feedback information. Accordingly, the feedback information input by the user in response to the interaction result of the first software not meeting the expectations can be received, wherein the feedback information is used to indicate the user's actual conversation intention.

[0151] Step S606: based on the feedback information and the dialogue information, re-determine the second software and the second interactive operation to be interacted.

[0152] The determining principle of the second software and the second interactive operation is similar to the determining principle of the first software and the second interactive operation, and will not be elaborated here.

[0153] Exemplarily, RAG technology can be used to retrieve similar software with similar semantics to the feedback information and dialogue information, and retrieve similar interfaces with similar semantics to the feedback information and dialogue information from multiple interactive interfaces associated with the similar software, call the large language model to determine the second software from the similar software based on the feedback information and dialogue information, determine the operation interface to be called from the similar interfaces, and use the interactive operation associated with the operation interface to be called as the second interactive operation.

[0154] Step S607: calling a second standard interaction component adapted to the second software and the second interaction operation from the standard interaction component library.

[0155] The calling method of the second standard interaction component is similar to the calling method of the first annotation interaction component, and will not be described in detail here.

[0156] Step S608: Display the second standard interaction component in the current dialog.

[0157] In the embodiment of the present application, the called second standard interaction component can be displayed in the current conversation of the conversation application, that is, the second standard interaction component is called out in the current conversation of the conversation application.

[0158] Step S609: Based on the second interaction information triggered by the second standard interaction component, a second interaction operation is performed on the second software.

[0159] It should be noted that the implementation principle of step S609 is similar to that of step S304, and will not be elaborated here.

[0160] As an example, the second standard interaction component includes an input interaction component and an output interaction component. It can obtain the second interaction information triggered by the user on the input interaction component displayed in the current conversation, and pass the second interaction information to the operation interface to be called, so as to perform a second interaction operation on the second software through the operation interface to obtain an interaction result, and display the interaction result through the output interaction component displayed in the current conversation.

[0161] The software interaction method based on AI, RPA and AI Agent in the embodiment of the present application, when the analysis and planning capabilities of the large language model are insufficient, re-calls the large language model to plan the call mapping process based on user feedback information, so as to interact with the software based on the re-planned call mapping process, thereby improving the accuracy of software interaction and improving the user experience in software interaction scenarios.

[0162] Changes in software technology often ultimately lead to changes in human-computer interaction methods. Conversely, smarter and more natural human-computer interaction methods have also been reshaping software. The natural language-based interaction method driven by large language models is reshaping almost all existing software and forming a new software development paradigm. This new paradigm, at the human-computer interaction level, takes natural language interaction as the core and thoroughly integrates various modes of interaction.

[0163] Among them, the traditional GUI can be more intelligently or naturally integrated with the dialogue and become a mode of dialogue. The main reasons include the following:

[0164] 1. In many cases, the efficiency of natural language dialogue itself is lower than that of GUI interface input. Formatting, visualization, and dynamic display are also more suitable for GUI. Therefore, dialogue interaction needs to be supplemented by GUI interaction.

[0165] 2. The powerful natural semantic understanding, generation and planning capabilities of the large language model also make GUI interaction lighter, simpler and more intuitive, and the interaction unfolds more naturally following the user's semantics;

[0166] 3. The analysis and planning capabilities of the large language model in related technologies are still insufficient. Specific GUI input can avoid the shortcomings of the large language model capabilities. The fusion process of GUI and dialogue interaction itself is also a process that can adapt to each other and develop.

[0167] In order to solve the problem of how to intelligently integrate GUI interaction into conversational interaction flow based on large language model, realize the interaction mode of natural integration of conversation and GUI, and how to quickly reconstruct the existing software functions into the natural integration interaction mode of conversation and GUI, drive the existing software distributed on a large number of devices of enterprises and individual users, realize the intelligent reshaping of the existing software system, and help enterprises and individuals quickly and gradually realize the new digital transformation, this application proposes a method such as Figure 7 The dialogue GUI shown in the figure is a fully intelligent fusion interactive implementation framework, and its main implementation process includes:

[0168] ① Define a set of standard, lightweight UI component libraries that are easy for large language models to understand, as the embedded standard interaction component library of the large language model;

[0169] ② Implement a local system UI and API automation proxy interface or tool to make the local interface call the underlying tool of the large language model;

[0170] ③ Using the large language model to implement the call mapping process of the underlying tools from UI interaction components to the local automation proxy interface;

[0171] ④Implement a conversational application driven by a large language model. Through the call mapping process generated by ③, call the UI interaction component from the standard interaction component library and call out the UI interaction component within the conversation.

[0172] The implementation framework of the fully intelligent fusion interaction of the dialogue GUI provided by this application has at least the following advantages over the related technologies:

[0173] 1. Based on the large language model, a set of interactive systems that integrate dialogue interaction and GUI interaction intelligence is defined and implemented, making GUI a natural interactive mode for dialogue-based intelligent agents and giving full play to the advantages of both.

[0174] 2. Define an intelligent interactive mapping and scheduling system that implements the fusion interaction of conversational GUI and the traditional GUI software across backend devices, so that the intelligent agent can quickly drive and reshape the software system in related technologies through conversational interaction and embedded GUI.

[0175] In order to implement the above embodiments, the present application also provides a software interaction device based on AI, RPA and AI Agent.

[0176] Figure 8 This is a structural diagram of a software interaction device based on AI, RPA and AI Agent provided by an embodiment of the present application.

[0177] like Figure 8As shown, the software interaction device 800 based on AI, RPA and AI Agent includes: an identification module 810, a calling module 820, a display module 830 and an execution module 840.

[0178] The identification module 810 is used to obtain the dialogue information from the dialogue application and identify the dialogue information to obtain the first software to be interacted and the first interactive operation;

[0179] A calling module 820 is used to call a first standard interaction component adapted to the first software and the first interaction operation from a standard interaction component library embedded in the large language model;

[0180] A display module 830, configured to display the first standard interaction component in a current conversation of the conversation application;

[0181] The execution module 840 is configured to execute a first interaction operation on the first software based on the first interaction information triggered by the first standard interaction component.

[0182] In any embodiment of the present application, the dialogue application is an application driven by a large language model, and the identification module 810 is used to: retrieve candidate software with semantically similar dialogue information from multiple interactive software; retrieve candidate interfaces with semantically similar dialogue information from multiple interactive interfaces associated with the candidate software; call the large language model to determine the first software from the candidate software and the target interface from the candidate interfaces based on the dialogue information; and use the interactive operation associated with the target interface as the first interactive operation.

[0183] In any embodiment of the present application, the identification module 810 is used to: when the candidate software includes an API interface, use the API interface as an interaction interface; when the candidate software does not include an API interface, use the RPA automation agent interface associated with the candidate software as an interaction interface; wherein the RPA automation agent interface is used to operate the user interface of the candidate software; and retrieve a candidate interface with semantics similar to the dialog information from the interaction interface.

[0184] In any embodiment of the present application, the identification module 810 is used to: obtain interface information and interface parameters of the candidate interface; generate prompt information based on the dialogue information, candidate software, interface information and interface parameters; wherein the prompt information is used to indicate the task information to be executed by the large language model; and process the prompt information using the large language model to obtain the first software and the target interface.

[0185] In any embodiment of the present application, the identification module 810 is used to: obtain interface information and interface parameters of the candidate interface; generate prompt information based on the dialogue information, candidate software, interface information and interface parameters; wherein the prompt information is used to indicate the task information to be executed by the large language model; and process the prompt information using the large language model to obtain the first software and the target interface.

[0186] In any embodiment of the present application, the identification module 810 is used to: use a large language model to process the prompt information to obtain a call mapping process output by the large language model; determine the first software to be interacted and the target interface from the call mapping process; wherein the call mapping process is used to indicate the mapping relationship between the first software to be interacted, the target interface in the candidate interface, the interaction operation associated with the target interface, and the interface parameters of the target interface.

[0187] In any embodiment of the present application, the prompt information also includes component information of multiple standard interaction components in the standard interaction component library; calling a mapping process, specifically for indicating a mapping relationship between the first software to be interacted, the target interface in the candidate interface, the interaction operation associated with the target interface, the interface parameters of the target interface, and the standard interaction components to be interacted in the standard interaction components;

[0188] The calling module 820 is used to: use the standard interaction component to be interacted indicated by the calling mapping process as the first standard interaction component; and call the first standard interaction component based on the interface parameters of the target interface indicated by the calling mapping process.

[0189] In any embodiment of the present application, the number of the first interaction operation is multiple, and the first standard interaction component includes an input interaction component and an output interaction component;

[0190] The execution module 840 is used to: obtain the first interaction information triggered by the input interaction component displayed in the current dialog; transmit the first interaction information to the target interface to perform multiple first interaction operations on the first software through the target interface to obtain an interaction result; and display the interaction result through the output interaction component displayed in the current dialog.

[0191] In any embodiment of the present application, there are multiple target interfaces; the execution module 840 is used to: obtain the calling order between the multiple target interfaces; based on the calling order between the multiple target interfaces, transmit the first interaction information to the multiple target interfaces in sequence; wherein the i-th target interface executes the first interaction operation having a mapping relationship with the i-th target interface on the first software based on the execution results of the previous i-1 target interfaces, the historical conversation information in the conversation application and the first interaction information; and the execution result of the last target interface is used as the interaction result.

[0192] In any embodiment of the present application, the software interaction device 800 based on AI, RPA and AI Agent further includes:

[0193] A receiving module, configured to receive feedback information input in response to an interaction result not meeting expectations;

[0194] A determination module, used to re-determine the second software and the second interactive operation to be interacted based on the feedback information and the dialogue information;

[0195] The calling module 820 is further used to: call a second standard interaction component adapted to the second software and the second interaction operation from the standard interaction component library;

[0196] The display module 830 is further used to: display the second standard interaction component in the current dialogue;

[0197] The execution module 840 is further configured to: execute a second interaction operation on the second software based on the second interaction information triggered by the second standard interaction component.

[0198] It should be noted here that the software interaction device based on AI, RPA and AI Agent provided in the embodiment of the present application can implement all the method steps implemented in any of the above-mentioned method embodiments, and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0199] Fig. 9 FIG. 1 is a block diagram showing a structure of an electronic device according to an embodiment of the present application. Fig. 9 As shown, the electronic device includes: a memory 910 and a processor 920, and the memory 910 stores a computer program that can be run on the processor 920. When the processor 920 executes the computer program, the software interaction method based on AI, RPA and AI Agent in the above embodiment is implemented. The number of the memory 910 and the processor 920 can be one or more.

[0200] The electronic device also includes:

[0201] The communication interface 930 is used to communicate with external devices and perform data exchange transmission.

[0202] If the memory 910, the processor 920 and the communication interface 930 are implemented independently, the memory 910, the processor 920 and the communication interface 930 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0203] Optionally, in a specific implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on a chip, the memory 910, the processor 920 and the communication interface 930 can communicate with each other through an internal interface.

[0204] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the software interaction method based on AI, RPA, and AI Agent provided in any embodiment of the present application.

[0205] An embodiment of the present application also provides a chip, which includes a processor for calling and running instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the software interaction method based on AI, RPA and AI Agent provided in any embodiment of the present application.

[0206] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the software interaction method based on AI, RPA and AIAgent provided in any embodiment of the application.

[0207] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting an advanced RISC machines (ARM) architecture.

[0208] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0209] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0210] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0211] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0212] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0213] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0214] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0215] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0216] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A software interaction method based on artificial intelligence AI, robotic process automation RPA and digital labor platform intelligent assistant AIAgent, characterized in that: include: Acquire the dialogue information from the dialogue application, and identify the dialogue information to obtain the first software to be interacted and the first interactive operation; Calling a first standard interaction component adapted to the first software and the first interaction operation from a standard interaction component library embedded in the large language model; Displaying the first standard interaction component in the current conversation of the conversation application; Based on the first interaction information triggered by the first standard interaction component, the first interaction operation is performed on the first software.

2. The method according to claim 1, characterized in that The dialog application is an application driven by the large language model, and the dialog information is identified to obtain the first software to be interacted with and the first interactive operation: Retrieving candidate software with semantic similarity to the dialogue information from multiple interactive software; Retrieving a candidate interface that is semantically similar to the dialog information from a plurality of interactive interfaces associated with the candidate software; Calling the large language model to determine the first software from the candidate software and determine the target interface from the candidate interfaces according to the dialog information; An interaction operation associated with the target interface is used as the first interaction operation.

3. The method according to claim 2, characterized in that The retrieving a candidate interface having semantic similarity with the dialog information from a plurality of interactive interfaces associated with the candidate software includes: In the case where the candidate software includes an API interface, using the API interface as the interactive interface; In the case where the candidate software does not include the API interface, an RPA automation agent interface associated with the candidate software is used as the interaction interface; wherein the RPA automation agent interface is used to operate the user interface of the candidate software; The candidate interface having semantic similarity to the dialog information is retrieved from the interactive interfaces.

4. The method according to claim 2, characterized in that: The calling of the large language model to determine the first software from the candidate software according to the dialog information, and determining the target interface from the candidate interfaces, includes: Obtaining interface information and interface parameters of the candidate interface; Generate prompt information according to the dialogue information, the candidate software, the interface information and the interface parameters; wherein the prompt information is used to indicate task information to be performed by the large language model; The prompt information is processed using the large language model to obtain the first software and the target interface.

5. The method according to claim 4, characterized in that The using the large language model to process the prompt information to obtain the first software and the target interface includes: Processing the prompt information using the large language model to obtain a call mapping process output by the large language model; Determining the first software and the target interface to be interacted from the call mapping process; The call mapping process is used to indicate the mapping relationship between the first software to be interacted, the target interface in the candidate interfaces, the interactive operation associated with the target interface, and the interface parameters of the target interface.

6. The method according to claim 5, characterized in that The prompt information also includes component information of multiple standard interaction components in the standard interaction component library; The call mapping process is specifically used to indicate the mapping relationship between the first software to be interacted, the target interface in the candidate interfaces, the interaction operation associated with the target interface, the interface parameters of the target interface, and the standard interaction components to be interacted in the standard interaction components; The calling of a first standard interaction component adapted to the first software and the first interaction operation from a standard interaction component library embedded in the large language model includes: Using the standard interaction component to be interacted indicated by the call mapping process as the first standard interaction component; The first standard interaction component is called based on the interface parameters of the target interface indicated by the call mapping process.

7. The method according to any one of claims 2 to 6, characterized in that: The number of the first interaction operations is multiple, and the first standard interaction component includes an input interaction component and an output interaction component; The performing the first interactive operation on the first software based on the first interactive information triggered by the first standard interactive component includes: Acquire first interaction information triggered by the input interaction component displayed in the current dialog; The first interaction information is transmitted to the target interface, so as to perform the plurality of first interaction operations on the first software through the target interface to obtain an interaction result; The interaction result is displayed through the output interaction component displayed in the current conversation.

8. The method according to claim 7, characterized in that The number of the target interfaces is multiple; The step of transmitting the first interaction information to the target interface so as to perform the plurality of first interaction operations on the first software through the target interface to obtain an interaction result includes: Obtaining a calling sequence between the plurality of target interfaces; Based on the calling order between the multiple target interfaces, the first interaction information is sequentially transmitted to the multiple target interfaces; wherein the i-th target interface performs the first interaction operation having a mapping relationship with the i-th target interface on the first software based on the execution result of the first i-1 target interfaces, the historical dialogue information in the dialogue application and the first interaction information; The last execution result of the target interface is used as the interaction result.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: receiving feedback information input in response to the interaction result not meeting expectations; re-determining the second software and the second interactive operation to be interacted based on the feedback information and the dialogue information; Calling a second standard interaction component adapted to the second software and the second interaction operation from the standard interaction component library; Displaying the second standard interaction component in the current dialog; Based on the second interaction information triggered by the second standard interaction component, the second interaction operation is performed on the second software.

10. A software interaction device based on AI, RPA and AIAgent, characterized in that: include: An identification module, used to obtain dialogue information from the dialogue application and identify the dialogue information to obtain the first software to be interacted and the first interaction operation; A calling module, configured to call a first standard interaction component adapted to the first software and the first interaction operation from a standard interaction component library embedded in a large language model; A display module, used for displaying the first standard interaction component in the current dialog of the dialog application; An execution module is used to execute the first interaction operation on the first software based on the first interaction information triggered by the first standard interaction component.

11. The device according to claim 10, characterized in that The dialogue application is an application driven by the large language model, and the recognition module is used to: Retrieving candidate software with semantic similarity to the dialogue information from multiple interactive software; Retrieving a candidate interface that is semantically similar to the dialog information from a plurality of interactive interfaces associated with the candidate software; Calling the large language model to determine the first software from the candidate software and determine the target interface from the candidate interfaces according to the dialog information; An interaction operation associated with the target interface is used as the first interaction operation.

12. The device according to claim 11, characterized in that The identification module is used to: In the case where the candidate software includes an API interface, using the API interface as the interactive interface; In the case where the candidate software does not include the API interface, an RPA automation agent interface associated with the candidate software is used as the interaction interface; wherein the RPA automation agent interface is used to operate the user interface of the candidate software; The candidate interface having semantic similarity to the dialog information is retrieved from the interactive interfaces.

13. The device according to claim 11, characterized in that The identification module is used to: Obtaining interface information and interface parameters of the candidate interface; Generate prompt information according to the dialogue information, the candidate software, the interface information and the interface parameters; wherein the prompt information is used to indicate task information to be performed by the large language model; The prompt information is processed using the large language model to obtain the first software and the target interface.

14. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 9.

15. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images