Deployment operation and maintenance agent construction method and device based on multi-modal large model
By building an internal and external double-layer structure intelligent body based on multimodal large model, the problems of time-consuming and labor-intensive manual operation and maintenance and limitations of automation tools in engineering deployment operation and maintenance are solved, and intelligent deployment operation and maintenance automation processing is realized, improving Efficiency and accuracy.
Patent Information
- Application Number
- CN202411507633.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing technology faces the needs of complex system architecture, frequent updates and iterations and uninterrupted service in engineering deployment and operation and maintenance. Manual operation and maintenance methods are time-consuming and labor-intensive and error-prone. Automation tools exist in exception handling, dynamic decision-making and cross-system collaboration. Limitations make it difficult to solve the problem of deployment operation and maintenance.
Using the deployment operation and maintenance agent construction method based on multimodal large models, RPA applications are built through operating system handles and multimodal large models, combined with search and enhancement generation model and standardized knowledge base, to build an internal and external dual-layer structure agent to realize Recognition of natural language input and task execution, and automated handling of abnormal situations.
Implement intelligent deployment operation and maintenance solutions that operate in various environments, improve deployment operation and maintenance efficiency and accuracy, reduce operation and maintenance costs, enhance the automation capabilities of exception handling, and solve traditional manual operation and maintenance and automation tools. Issues of limitations.
Smart Images

Figure CN119067158A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for constructing a deployment and operation and maintenance intelligent entity based on a multimodal large model. Background Art
[0002] In the current engineering deployment and operation and maintenance field, the usual solution is to rely on manual operation and maintenance, or use some automated tools. However, in the face of increasingly complex system architecture, frequent updates and iterations, and uninterrupted service requirements, traditional manual operation and maintenance methods are facing huge challenges. Manual operations are not only time-consuming and labor-intensive, but also prone to errors, resulting in service interruptions or performance degradation.
[0003] Although automation tools such as CI / CD (continuous integration / continuous deployment) and configuration management software (such as Ansible and Terraform) have improved deployment and operation efficiency to a certain extent, they still have limitations in terms of exception handling, dynamic decision-making, and cross-system collaboration.
[0004] This shows that the deployment and operation and maintenance systems (intelligent agents) in related technologies have technical problems with large limitations in deployment and operation and maintenance. Summary of the invention
[0005] The present invention provides a method and device for constructing a deployment and operation and maintenance intelligent body based on a multimodal large model, which is used to solve the defect that the deployment and operation and maintenance system in the related technology in the prior art has large limitations on deployment and operation, and realizes the construction of an intelligent body with an internal and external double-layer structure for engineering deployment and operation and maintenance scenarios.
[0006] The present invention provides a method for constructing a deployment and operation intelligent agent based on a multimodal large model, comprising the following steps. Construct an RPA application based on an operating system handle and a multimodal large model, wherein the multimodal large model is used for image recognition and text recognition; perform permutations and combinations based on the atomic capabilities of the RPA application to obtain multiple RPA tool chains, wherein each of the multiple RPA tool chains is used to perform different target tasks; obtain a preset retrieval enhancement generation model and a preset standardized knowledge base document; input the standardized knowledge base document into the retrieval enhancement generation model to obtain a knowledge memory and retrieval module, and construct an outer brain of a deployment and operation intelligent agent based on the long and short-term memory of the knowledge memory and retrieval module and the preset large language model, wherein the outer brain is used to recognize natural language input; divide the multiple RPA tool chains into multiple task executors according to the target tasks, and construct an inner brain of the deployment and operation intelligent agent based on the multiple task executors, wherein the inner brain is used to call the inner task executor to execute an executable task flow corresponding to the natural language input.
[0007] According to a method for constructing a deployment and operation and maintenance intelligent agent based on a multimodal big model provided by the present invention, the RPA application is constructed based on an operating system handle and a multimodal big model, including: obtaining an operating system handle and a preset multimodal big model; in a local environment, based on the operating system handle, determining the basic capabilities of the RPA application, wherein the basic capabilities include at least one of the following: input simulation, process control, result output, and system interaction; in a remote environment, based on the multimodal big model, determining the composite capabilities of the RPA application, wherein the composite capabilities include at least one of the following: intent analysis and report analysis.
[0008] According to a method for constructing a deployment and operation and maintenance intelligent agent based on a multimodal large model provided by the present invention, before obtaining a preset retrieval enhancement generation model and a preset standardized knowledge base document, the method also includes: obtaining standard training data, wherein the standard training data includes a standardized process of deployment and operation, a historical interactive feedback process, and historical interactive feedback results; vectorizing the standard training data to obtain a training data vector; storing the training data vector in a vector database; and constructing a standardized knowledge base document based on the vector database.
[0009] According to a method for constructing a deployment and operation and maintenance intelligent agent based on a multimodal large model provided by the present invention, the method also includes: obtaining the natural language input; retrieving and identifying the natural language input through the outer brain of the deployment and operation and maintenance intelligent agent to obtain user intention; splitting the user intention through the outer brain of the deployment and operation and maintenance intelligent agent to obtain at least one executable task flow and the execution order of the executable task flow.
[0010] According to a method for constructing a deployment and operation and maintenance intelligent body based on a multimodal large model provided by the present invention, after splitting the user intention to obtain at least one executable task flow and the execution order of the executable task flow, the method also includes: determining a plurality of inner-layer task executors based on the executable task flow and the execution order through the inner-layer brain of the deployment and operation and maintenance intelligent body; calling the plurality of inner-layer task executors to execute the executable task flow in the execution order through the inner-layer brain of the deployment and operation and maintenance intelligent body to obtain an interactive feedback process and interactive feedback results.
[0011] According to a method for constructing a deployment and maintenance intelligent agent based on a multimodal large model provided by the present invention, the method also includes: when the interactive feedback process is inconsistent with the executable task flow, determining the abnormal result through the inner brain of the deployment and maintenance intelligent agent; calling the inner task executor through the inner brain of the deployment and maintenance intelligent agent to feed back the abnormal result to the outer brain of the deployment and maintenance intelligent agent; searching the standardized knowledge base document based on the abnormal result and long and short-term memory through the outer brain of the deployment and maintenance intelligent agent; when a scenario solution corresponding to the abnormal result is retrieved, the executable task flow is searched through the outer brain of the deployment and maintenance intelligent agent. The method comprises the following steps: completing the abnormal result to obtain a supplementary task flow command; and sending the supplementary task flow command to the inner task executor of the inner brain of the deployment and operation and maintenance intelligent body; when a scenario solution corresponding to the abnormal result is not retrieved and the abnormal result hinders the subsequent execution process of the executable task flow, the abnormal result is fed back to the user through the outer brain of the deployment and operation and maintenance intelligent body; in response to obtaining a feedback instruction for the abnormal result input by the user; sending the feedback instruction to the inner task executor of the inner brain of the deployment and operation and maintenance intelligent body through the outer brain of the deployment and operation and maintenance intelligent body, and storing the abnormal processing flow corresponding to the abnormal result in the retrieval enhancement generation model.
[0012] The present invention also provides a deployment and operation and maintenance intelligent body construction device based on a multimodal large model, comprising the following modules: a construction module, used to construct an RPA application based on an operating system handle and a multimodal large model, wherein the multimodal large model is used for image recognition and text recognition; a combination module, used to perform permutations and combinations based on the atomic capabilities of the RPA application to obtain multiple RPA tool chains, wherein each of the multiple RPA tool chains is used to perform different target tasks; an acquisition module, used to acquire a preset retrieval enhancement generation model and a preset standardized knowledge base document; a retrieval module, used to obtain ... The standardized knowledge base document is input into the retrieval enhancement generation model to obtain a knowledge memory and retrieval module, and based on the long and short-term memory of the knowledge memory and retrieval module and the preset large language model, an outer brain of the deployment and operation and maintenance intelligent body is constructed, wherein the outer brain is used to recognize natural language input; a division module is used to divide the multiple RPA tool chains into multiple task executors according to the target tasks, and an inner brain of the deployment and operation and maintenance intelligent body is constructed based on the multiple task executors, wherein the inner brain is used to call the inner task executor to execute the executable task flow corresponding to the natural language input.
[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a method for constructing a deployment and operation intelligent entity based on a multimodal large model as described in any one of the above.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for constructing a deployment and operation intelligent agent based on a multimodal large model as described in any one of the above.
[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for constructing a deployment and operation and maintenance intelligent agent based on a multimodal large model.
[0016] The method and device for constructing a deployment and operation and maintenance intelligent body based on a multimodal big model provided by the present invention construct an RPA application based on an operating system handle and a multimodal big model. Therefore, when the RPA application is running on the local machine, it can execute the local workflow through the operating system handle. When the RPA application is running in a remote environment, the handle operation is replaced by multimodal technology to complete the remote workflow. The atomic capabilities of the RPA application are arranged and combined to obtain an RPA tool chain for executing different target tasks respectively. The standardized knowledge base document is input into the retrieval enhancement generation model to obtain a knowledge memory and retrieval module. Based on the knowledge memory and retrieval module and the preset big language, the RPA tool chain is constructed. The long-term and short-term memory of the language model is used to build the outer brain of the deployment and operation and maintenance intelligent body for recognizing natural language input. As a result, the outer brain can recognize user intentions based on user input; multiple RPA tool chains are divided into multiple task executors according to target tasks to build the inner brain of the deployment and operation and maintenance intelligent body. As a result, the target task executor can be called through the inner brain to execute the executable task flow corresponding to the user intention; and then an inner-outer double-layer structure intelligent body for engineering deployment and operation and maintenance scenarios that can run in a variety of environments is constructed, which solves the technical problem that the deployment and operation and maintenance system (intelligent body) in related technologies has large limitations in deployment and operation. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 It is a flow chart of the method for constructing a deployment and operation and maintenance intelligent body based on a multimodal large model provided by the present invention.
[0019] Figure 2 It is a structural diagram of the deployment and operation and maintenance intelligent body provided by the present invention.
[0020] Figure 3 It is a flow chart of construction and adaptation of the retrieval enhancement generation model provided by the present invention.
[0021] Figure 4 It is a flow chart of the application of the solution for deploying operation and maintenance intelligent bodies provided by the present invention.
[0022] Figure 5 It is a structural diagram of a deployment and operation and maintenance intelligent body construction device based on a multimodal large model provided by the present invention.
[0023] Figure 6 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] In the current engineering deployment and operation and maintenance field, the usual solution is to rely on manual operation and maintenance, or with the help of some automated tools. However, faced with increasingly complex system architectures, frequent updates and iterations, and 7 days / 24 hours of uninterrupted service requirements, traditional manual operation and maintenance methods are facing huge challenges. Manual operations are not only time-consuming and labor-intensive, but also prone to errors, resulting in service interruptions or performance degradation. Although automated tools such as CI / CD (continuous integration / continuous deployment) and configuration management software (such as Ansible and Terraform) have improved deployment and operation efficiency to a certain extent, they still have limitations in terms of exception handling, dynamic decision-making, and cross-system collaboration.
[0026] In the related art, a method for optimizing RPA operation is provided. The method can continuously obtain operation information during the operation of the RPA program; by comparing the operation information with the configuration information, in the case of inconsistency, an error message is generated according to the operation information, and the corresponding emergency operation is performed, so that the RPA process can run smoothly.
[0027] However, in the above technology, only the problems encountered during the RPA execution process are automatically corrected, but the RPA workflow itself still needs to be manually written by the user; after the RPA workflow is executed, there is no automated solution for problems that arise in the business results. This solution is only applicable to the optimization of the RPA workflow execution process, but does not process the business results after the RPA execution is completed. It is not applicable to solving various problems that arise during the deployment and operation process.
[0028] In the related technology, a solution for the operation and maintenance system of the power industry is provided. The method realizes dialect recognition based on Python language, converts user voice input into text output; then analyzes the output text in combination with a large language model to screen out standard questions; then retrieves answers from the knowledge base to answer the corresponding questions or transfers them to the corresponding contact personnel; finally, the efficiency of connecting and solving user problems in the power industry is improved.
[0029] However, the above technologies provide more optimization solutions for the customer service scenario in the power industry. Based on the large language model, it can quickly analyze the user's intention during the user's questioning process, locate the accurate classification of the user's question, search for similar cases in the knowledge base, provide approximate answers or transfer the call to the operator in the accurate classification. This solution does not belong to the traditional deployment and operation solution for the application itself; at the same time, this solution is not suitable for solving various problems that arise during the deployment and operation process.
[0030] In the related technology, a digital employee solution for enterprises is provided to improve the deployment and operation efficiency of service platforms, the application effect of service platforms, and the interaction efficiency between users and service platforms. The method uses RPA technology to obtain the business-related data and workflow data of the enterprise, and then uses the artificial intelligence AI platform to process the business-related data to obtain structured target business data. According to the target business data and workflow data, the corresponding processing logic corresponding to the enterprise can be generated to control the digital employee execution device.
[0031] However, among the above technologies, the more inclined one is to provide enterprises with a digital employee solution. Enterprises obtain the business and workflow data of the enterprise through predefined RPA methods, and then process these data with the help of AI capabilities, and then automatically execute the corresponding device content; in the middle, NLP technology is combined to process the text feedback given by external users during the execution of digital employees, to help improve the interaction effect. This solution is also not a traditional deployment and operation solution for the application itself; at the same time, this solution is not suitable for solving various problems that arise during the deployment and operation process.
[0032] To sum up, the current technical solutions are unable to solve the various problems that arise during the deployment, use, inspection, and operation and maintenance of high-speed iterative software applications in different deployment environments.
[0033] In the project deployment and operation scenario, in addition to the above problems, the following problems also exist.
[0034] In some private deployment environments, usually LAN environments, public network access capabilities are limited. Traditional deployment, operation and maintenance methods, as well as inspection and alarm solutions, are not applicable to private deployment scenarios here, resulting in low deployment efficiency, high operation and maintenance costs, and failure to discover and transmit faults in a timely manner. At the same time, traditional Robotic Process Automation (RPA) technology relies on obtaining operating system handles to execute different workflows. For these private deployment scenarios, after entering the remote system, RPA cannot obtain the corresponding handle, resulting in the inability to execute the workflow in normal order.
[0035] At the same time, with the rapid development of artificial intelligence technology, especially the emergence of large language models, new possibilities have been provided for solving the above problems. Large language models have shown strong capabilities in understanding natural language, generating code snippets, and providing decision-making suggestions, but how to effectively apply them to engineering deployment and operation and maintenance remains a major challenge.
[0036] The purpose of this invention is to build an artificial intelligence (AI) agent for deployment and operation scenarios. This agent combines language models, multimodal large models, RPA and other technologies to optimize the traditional engineering deployment and operation processes, solve the various problems in the above-mentioned deployment and operation scenarios, and provide users with a more intelligent deployment and operation solution.
[0037] Optionally, the method for constructing a deployment and operation and maintenance intelligent body based on a multimodal big model in an embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the example of the method for constructing a deployment and operation and maintenance intelligent body based on a multimodal big model in this embodiment being executed by a server.
[0038] Figure 1 It is a flow chart of the method for constructing a deployment and operation and maintenance intelligent agent based on a multimodal large model provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 101, building an RPA application based on an operating system handle and a multimodal large model, wherein the multimodal large model is used for image recognition and text recognition.
[0039] In an embodiment of the present invention, based on an open source RPA application, an RPA application suitable for deployment and operation is constructed based on operating system handles and multimodal text and image recognition, so as to solve the problem that the public network is not accessible and the handle cannot be obtained in a private deployment environment (remote environment).
[0040] Here, a handle in the Windows operating system is a 32-bit unsigned integer value that represents a structure pointer in the operating system's memory. This structure contains various information related to the object, such as properties, status, etc. The operating system manages and accesses these objects through handles rather than directly through the object's memory address.
[0041] Robotic Process Automation (RPA) automates a series of preset, rule-based tasks and workflows by simulating the operation behavior of human users on a computer interface. RPA technology uses software robots to perform these tasks. These robots can recognize, trigger and interact with user interface elements, such as opening applications, entering data, clicking buttons, etc.
[0042] In an embodiment of the present invention, the constructed RPA application is deployed in a local environment (local machine). When the RPA application runs on the local machine, the corresponding workflow can be executed by obtaining the operating system handle; when the RPA application runs in a remote (privatized) environment, multimodal technology (ordinary image OCR, combined with multimodal large model capabilities) is used to rely on methods such as image and text recognition to perform intelligent analysis on the results of the RPA application execution, including but not limited to report analysis, intent analysis, etc., thereby completing the relevant workflow instead of the handle operation.
[0043] Step 102: Arrange and combine the atomic capabilities of the RPA application to obtain multiple RPA tool chains, wherein each of the multiple RPA tool chains is used to execute different target tasks.
[0044] In the embodiment of the present invention, the capabilities of the RPA application constructed in the above step 101 are classified and extracted to form composite capabilities and basic capabilities. These capabilities are arranged and combined based on the atomic capabilities of the RPA application, thereby forming a series of RPA tools for use by AI-Agents. In addition to the atomic capabilities of RPA itself, user customization is also supported, such as third-party SDK calls, etc., to expand the RPA application capabilities through user customization. For specific RPA capabilities, please refer to Figure 2 The part below the dotted line (AIRPA).
[0045] Among them, AI Agent is a system driven by a large language model as its brain. It has the ability to autonomously understand, perceive, plan, remember and use tools, and can automatically execute and complete complex tasks.
[0046] Step 103: Obtain a preset retrieval enhancement generation model and a preset standardized knowledge base document.
[0047] In an embodiment of the present invention, a retrieval-augmented generation (RAG) model combines a language model and information retrieval technology. When the retrieval-augmented generation model needs to generate text or answer a question, it first retrieves relevant information from a document collection (i.e., a standardized knowledge base document), and then uses the retrieved information to guide the generation of text, thereby improving the quality and accuracy of the prediction.
[0048] The RAG model works through three key parts: retrieval, utilization, and generation. In the retrieval stage, the system retrieves relevant information from the document collection; in the utilization stage, the system uses the retrieved information to fill in the text or answer questions; and finally in the generation stage, the system generates the final text content based on the retrieved knowledge.
[0049] Step 104, input the standardized knowledge base document into the retrieval enhancement generation model to obtain the knowledge memory and retrieval module, and build the outer brain of the deployment operation and maintenance intelligent body based on the long and short-term memory of the knowledge memory and retrieval module and the preset large language model, wherein the outer brain is used to recognize natural language input.
[0050] In the embodiment of the present invention, it is necessary to build an internal and external double-layer loop deployment and operation intelligent entity.
[0051] Among them, the outer layer uses a large language model to act as the brain of the intelligent agent. It combines various standardized knowledge base documents accumulated through deployment and operation, and inputs them into the preset retrieval enhancement generation model to form a knowledge memory and retrieval module for the intelligent agent's brain. Combined with the long and short memory capabilities of the intelligent agent's brain, the outer layer brain is constructed.
[0052] At the same time, the outer brain also acts as a semantic recognition capability, which can identify the user's natural language input, reasonably split the user's intention into executable workflows, and pass them to the task planner; and plan the executable task flow through the task planner to obtain the execution order of the executable task flow.
[0053] Step 105: divide multiple RPA tool chains into multiple task executors according to target tasks, and build an inner brain for deploying the operation and maintenance intelligent body based on the multiple task executors, wherein the inner brain is used to call the inner task executors to execute the executable task flow corresponding to the natural language input.
[0054] In the embodiment of the present invention, the inner brain of the deployment operation and maintenance agent is a task executor, which includes different RPA tool chains and small parameter models, and executes related tasks in series or in parallel according to the task flow sequence planned by the outer layer, and finally completes the user's intention. The different capabilities of the RPA tool chain can be planned as different executors for use by the inner brain of the agent.
[0055] Through the above steps of the embodiment of the present invention, an RPA application is constructed based on the operating system handle and the multimodal large model, wherein the multimodal large model is used for image recognition and text recognition; multiple RPA tool chains are obtained by permutation and combination based on the atomic capabilities of the RPA application, wherein each RPA tool chain in the multiple RPA tool chains is used to execute different target tasks respectively; a preset retrieval enhancement generation model and a preset standardized knowledge base document are obtained; the standardized knowledge base document is input into the retrieval enhancement generation model to obtain a knowledge memory and retrieval module, and based on the knowledge memory and retrieval module and the long and short-term memory of the preset large language model, an outer brain of the deployment operation and maintenance intelligent body is constructed, wherein the outer brain is used to recognize natural language input; multiple RPA tool chains are divided into multiple task executors according to the target tasks, and an inner brain of the deployment operation and maintenance intelligent body is constructed based on the multiple task executors, wherein the inner brain is used to call the inner task executor to execute the executable task flow corresponding to the natural language input. Furthermore, we built an intelligent agent with an internal and external double-layer structure that can operate in a variety of environments for engineering deployment and operation scenarios, solving the technical problem that the deployment and operation of the operation system (intelligent agent) in related technologies has great limitations in deployment and operation.
[0056] According to a method for constructing a deployment and operation agent based on a multimodal large model provided by the present invention, an RPA application is constructed based on an operating system handle and a multimodal large model, including: Get the operating system handle and the preset multimodal large model; In the local environment, based on the operating system handle, determine the basic capabilities of the RPA application, where the basic capabilities include at least one of the following: input simulation, process control, result output, and system interaction; In a remote environment, based on a multimodal large model, determine the composite capabilities of the RPA application, where the composite capabilities include at least one of the following: intent analysis and report analysis.
[0057] In an embodiment of the present invention, based on an open source RPA application, an RPA application suitable for deployment and operation is constructed based on operating system handles and multimodal text and image recognition, so as to solve the problem that the public network is not accessible and the handle cannot be obtained in a private deployment environment.
[0058] The RPA application is deployed on the local machine. When the RPA application runs on the local machine, the corresponding workflow (corresponding to the executable task flow mentioned above) can be executed by obtaining the operating system handle. When the RPA application runs in a remote (privatized) environment, multimodal technology (ordinary image OCR, combined with multimodal large model capabilities) is used to rely on methods such as image and text recognition to perform intelligent analysis on the results of the RPA application execution, including but not limited to report analysis, intent analysis, etc., thereby completing the relevant workflow instead of the handle operation.
[0059] The capabilities of the built RPA applications are classified and extracted to form composite capabilities and basic capabilities. These capabilities are arranged and combined based on the atomic capabilities of the RPA application, thus forming a series of RPA tools for AI-Agents (intelligent entities) to use. In addition to the atomic capabilities of RPA itself, it also supports user customization, such as third-party SDK calls, etc., to expand RPA application capabilities through user customization. Specific RPA capability reference Figure 2 The part below the dotted line (AI RPA).
[0060] refer to Figure 2 , Figure 2 It is a structural diagram of the deployment and operation and maintenance intelligent body provided by the present invention, which includes: an inner brain (i.e., AI RPA) and an outer brain (i.e., AI Agent).
[0061] like Figure 2 As shown, the inner brain (i.e. AI RPA) includes basic capabilities and complex capabilities.
[0062] Basic capabilities include process control (sequential execution, loop execution, process jump and result output), input simulation (keyboard keys, mouse keys, mouse wheel, mouse movement, voice input and gesture input), result output (document generation, IM operation, Office processing and notification sending) and system interaction (screen capture, application startup, application shutdown, browser operation, OCR acquisition and clipboard operation); composite capabilities include official Action (platform login, report sending, result screenshot and multimodal analysis) and custom Action (platform login).
[0063] like Figure 2 As shown, the outer brain (i.e., AI Agent) includes the application layer and the large model AI-Agent.
[0064] The application layer includes the Agent client (local monitoring), and the large model AI-Agent includes structure judgment, semantic understanding, RAG module-knowledge retrieval / memory, exception handling and arrangement completion, and Action calling.
[0065] By using the embodiments provided by the present invention, an RPA application is constructed. When running in a local environment, the corresponding workflow can be executed by obtaining an operating system handle. When running in a remote environment, the results of the RPA application execution can be intelligently analyzed through multimodal technology. This solves the problem of a public network being unavailable and a handle being unavailable in a private deployment environment.
[0066] According to a method for constructing a deployment and operation agent based on a multimodal large model provided by the present invention, before obtaining a preset retrieval enhancement generation model and a preset standardized knowledge base document, the method further includes: Obtain standard training data, where the standard training data includes standardized processes for deployment and operation, historical interactive feedback processes, and historical interactive feedback results; Vectorize the standard training data to obtain a training data vector; Storing training data vectors in a vector database; Build standardized knowledge base documents based on vector database.
[0067] In an embodiment of the present invention, a full database is constructed based on relevant documents of deployment and operation, and the data is vectorized and saved using RAG technology. After analyzing the user's intention, the target content can be retrieved from the relevant documents and returned to the retrieval enhancement generation model. The target content is used as standard training data. The target content includes similar task workflows, solutions and processes for related exceptions.
[0068] Fixed system deployment and operation and maintenance processes are organized into standard processes as standard training data for deployment and operation and maintenance agents to learn and store, so that when performing similar tasks in the future, the agents can quickly and correctly plan related workflows.
[0069] When the operation and maintenance agent is deployed to perform related tasks and interact with users for feedback, the corresponding data can be stored as subsequent training fine-tuning data (i.e., standard training data).
[0070] refer to Figure 3 , Figure 3 It is a flow chart of the construction and adaptation of the retrieval enhancement generation model provided by the present invention, wherein the construction and adaptation of the retrieval enhancement generation model include data indexing, retrieval and generation; data indexing includes data extraction, embedding and index creation; generation includes automatic sorting of retrieval results and large language model (LLM) induction.
[0071] Based on the standard training data obtained above, the embodiment of the present invention can further construct and adapt the RAG, and the process includes the following steps: Figure 3 shown.
[0072] Data extraction: All standard training data are processed uniformly according to the required format. Then, by configuring the parameters required by the document segmentation algorithm and the vectorization model parameters, all standard training data will be extracted as training data vectors and stored in the vector database after program processing.
[0073] In the embodiment of the present application, all training data are preprocessed in the required format to ensure the consistency and accuracy of the data. The parameters of the document segmentation algorithm (such as TF-IDF, BERT, etc.) and the parameters of the vectorization model (such as Word2Vec, BERT Embedding, etc.) are configured as required. The preprocessed data is converted into vectors through the vectorization model, and these vectors can capture the semantic information of the data. The converted vectors are stored in the vector database for subsequent rapid retrieval.
[0074] In the embodiment of the present invention, a standardized knowledge base document is constructed based on a vector database.
[0075] Data retrieval: By configuring the reordering model and parameters and combining it with the RAG model, you can retrieve data content that meets the user's question from the vector database in the previous step.
[0076] In the embodiment of the present application, the re-ranking model (such as BM25, deep learning model, etc.) and its parameters are configured according to the requirements to rank the preliminary search results. The retrieval module in the RAG model is used in combination with the configured re-ranking model to retrieve the data content most relevant to the user's question from the vector database. The retrieved data content is used as a candidate set and prepared for the subsequent data generation step.
[0077] Data generation: Through prompt engineering techniques, a prompt word template is designed, which will guide the large language model to give an inductive answer through a chain of thought, and then output it to the user. At the same time, the user's input and the content retrieved in the previous step need to be put into the prompt word template to obtain a complete prompt word.
[0078] In an embodiment of the present invention, a prompt word template is designed through prompt engineering techniques to guide a large language model (such as GPT, BERT, etc.) to give an inductive answer through a chain of thoughts. The user's input and the retrieved content are put into the prompt word template to obtain a complete prompt word. This prompt word will be used as the input of the large language model. The complete prompt word is input into the large language model, and the model generates an inductive answer based on the prompt word and outputs it to the user.
[0079] Through the embodiments of the present invention, standard data in the deployment and operation and maintenance process, the historical interaction feedback process of the intelligent agent, and the historical interaction feedback results can be obtained as standard training data, so that when performing similar tasks in the future, the intelligent agent can quickly and correctly plan the relevant workflow.
[0080] According to a method for constructing a deployment and operation agent based on a multimodal large model provided by the present invention, the method further includes: Get natural language input; By deploying the outer brain of the operation and maintenance agent, natural language input is retrieved and recognized to obtain user intent; By deploying the outer brain of the operation and maintenance agent, the user intention is split to obtain at least one executable task flow and the execution order of the executable task flow.
[0081] In an embodiment of the present invention, a large language model is used as the outer brain of the intelligent agent, and various standardized knowledge base documents accumulated through deployment and operation are input into a preset retrieval enhancement generation model to form a knowledge memory and retrieval module for use by the intelligent agent's brain. The outer brain is constructed in combination with the long and short memory capabilities of the intelligent agent's brain.
[0082] At the same time, the outer brain also acts as a semantic recognition capability, which can identify the user's natural language input, reasonably split the user's intention corresponding to the natural language input into executable workflows, and pass them to the task planner; and plan the executable task flow through the task planner to obtain the execution order of the executable task flow.
[0083] In an embodiment of the present invention, a large language model (such as the GPT series, BERT, etc.) is used as the core processing unit of the outer brain of the intelligent agent, which is responsible for understanding, reasoning and decision-making. Various standardized knowledge base documents accumulated during the operation and maintenance process are input into the retrieval enhancement generation model to form a knowledge memory and retrieval module of the outer brain of the intelligent agent. This module not only provides knowledge retrieval functions, but also generates and reasons knowledge according to the context. The long short-term memory (LSTM) of the large language model itself or the self-attention mechanism in the Transformer architecture is used to realize the memory and forgetting of historical information by the intelligent agent, so as to better adapt to the dynamically changing operation and maintenance environment. The outer brain understands the user's natural language input through semantic recognition technology and splits its intention into a series of executable workflows. These workflows are then passed to the task planner for further scheduling and execution.
[0084] Through the embodiments of the present invention, the outer brain acts as a semantic recognition capability, which can recognize the user's natural language input, reasonably split the user's intention into executable workflows, and pass them to the task planner to obtain a more efficient execution order of the executable task flow.
[0085] According to a method for constructing a deployment and operation agent based on a multimodal large model provided by the present invention, after splitting the user intention to obtain at least one executable task flow and the execution order of the executable task flow, the method further includes: By deploying the inner brain of the operation and maintenance agent, multiple inner task executors are determined based on the executable task flow and execution order; By deploying the inner brain of the operation and maintenance intelligent body, multiple inner task executors are called to execute the executable task flow in order to obtain the interactive feedback process and interactive feedback results.
[0086] In the embodiment of the present invention, the inner brain of the deployment operation and maintenance intelligent body is a task executor, which includes different RPA tool chains and small parameter models, and executes related tasks in series or in parallel according to the task flow sequence planned by the outer layer, and finally completes the user's intention and obtains the interactive feedback process and interactive feedback results. The different capabilities of the RPA tool chain can be planned as different executors for use by the inner brain of the intelligent body.
[0087] In the embodiment of the present invention, the inner task executor is the core execution unit of the inner brain of the agent, which is responsible for calling different tool chains and small parameter models according to the executable task flow planned by the outer brain, and executing related tasks in series or in parallel to finally complete the user's intention. The different capabilities of RPA (Robotic Process Automation) can be planned as different task executors for use by the inner brain of the agent.
[0088] Through the embodiments of the present invention, the inner brain of the intelligent body can call the inner task executor to execute the executable task flow sent by the outer brain to complete the user's intention and provide feedback.
[0089] According to a method for constructing a deployment and operation agent based on a multimodal large model provided by the present invention, the method further includes: When the interactive feedback process does not match the executable task flow, determining an abnormal result through the inner brain of the deployment and operation intelligent agent; Through the inner brain of the deployment and operation and maintenance intelligent body, the inner task executor is called to feed back the abnormal result to the outer brain of the deployment and operation and maintenance intelligent body; Through the outer brain of the deployed operation and maintenance intelligent agent, searching the standardized knowledge base document based on the abnormal results and long and short-term memory; When a scenario solution corresponding to the abnormal result is retrieved, the executable task flow is completed through the outer brain of the deployment and operation and maintenance intelligent body to obtain a supplementary task flow command; and the supplementary task flow command is sent to the inner task executor of the inner brain of the deployment and operation and maintenance intelligent body; When no scenario solution corresponding to the abnormal result is retrieved and the abnormal result hinders the subsequent execution process of the executable task flow, the abnormal result is fed back to the user through the outer brain of the deployment and operation agent; In response to obtaining a feedback instruction input by the user for the abnormal result; The feedback instruction is sent to the inner task executor of the inner brain of the deployment and maintenance intelligent body through the outer brain of the deployment and maintenance intelligent body, and the exception handling process corresponding to the abnormal result is stored in the retrieval enhancement generation model.
[0090] In the embodiment of the present invention, when the inner brain of the operation and maintenance intelligent body is deployed and the inner task executor is called to execute the executable workflow (RPA execution), if the interactive feedback process or the interactive feedback result is inconsistent with the executable workflow planned by the outer brain, the abnormal result will be fed back to the inner task executor (that is, the inner task executor determines the abnormal result), and the inner task executor will further feed back to the outer brain. The outer brain makes the following two judgments based on historical experience: Combined with long and short memory, relevant scenario solutions are retrieved in the standardized knowledge base document (RAG knowledge base). If a hit is found, the current workflow logic is completed and a command is sent to the inner task executor to continue the next task.
[0091] If no corresponding scenario is retrieved and it is judged to be an exception that seriously hinders the current workflow, it will be displayed on the client and fed back to the user. The user interacts with the agent through the page on the client to confirm the next processing action. After receiving the user feedback, the outer brain sends instructions to the inner task executor according to the actual feedback to continue or terminate; the inner task executor strictly executes according to the outer brain's command. At the same time, the outer brain will automatically record the abnormal process processing in its own RAG module (standardized knowledge base document).
[0092] After going through the above implementation examples, an agent process based on RPA and multimodal large models has ended. The outer brain will remember this process, and when the user describes a similar requirement next time, the brain will automatically combine the memory and quickly build a correct and complete workflow based on past experience, thereby improving efficiency and user experience.
[0093] refer to Figure 4 , Figure 4It is a solution application flow chart for deploying operation and maintenance intelligent bodies provided by the present invention, which specifically includes the following steps.
[0094] In step 1, the user inputs the task through natural language.
[0095] In step 2, the agent generates a work step (RPA-Action).
[0096] In step 3, the user confirms the work step.
[0097] In step 4, the user issues an execution command to the agent.
[0098] In step 5, the agent calls the workflow (Action) to Robotic Process Automation (RPA).
[0099] In step 6, the Robotic Process Automation executes the workflow (Action).
[0100] In step 7, the computer (PC) feeds the results back to the robotic process automation (RPA).
[0101] In step 8, the Robotic Process Automation (RPA) feeds the results back to the agent.
[0102] In step 9, the agent performs feedback correction (adding an exception step).
[0103] In step 10, the agent completes the execution and obtains the work result.
[0104] In step 11, the working results are processed by the result algorithm, and the (final) working results are obtained and input into other models.
[0105] In step 12, the result is output to the user.
[0106] Among them, the workflow is generated through multiple rounds of dialogue through steps 1 / 2 / 3, and the Robotic Process Automation (RPA) is called multiple times in the order of the workflow through steps 5 / 6 / 7 / 8 / 9.
[0107] In summary, when users perform deployment and operation-related work, they only need to input work tasks through natural language. The agent can autonomously break down the steps and convert them into corresponding RPA execution actions. After the user confirms the execution action, the agent can call RPA-Action (the minimum atomic capability set of RPA) to complete the work content expected by the user. During the execution process, if an abnormal situation is encountered, the agent can automatically add the abnormal steps to the work steps and complete related work such as feedback correction. For related work steps that have been executed and confirmed by the user, after the user mentions it again, the agent can autonomously output and execute the correct content by memorizing historical experience.
[0108] This paper optimizes the open source RPA technology and combines AI capabilities to build an AI-RPA that can run in multiple environments. It can run in a local environment or in a private deployment scenario. It can provide both atomic capabilities and composite capabilities to register with the AI agent for scheduling.
[0109] The present invention leverages the powerful human-brain capabilities of a large language model to construct an AI agent with an internal and external double-layer structure for engineering deployment and operation and maintenance scenarios. It also combines mainstream technologies such as RAG to build a historical experience plug-in knowledge base, and then uses self-developed AI-RPA technology that can run freely in various environments to solve the problems in traditional deployment and operation and maintenance scenarios.
[0110] The following describes the deployment and operation and maintenance intelligent body construction device based on the multimodal big model provided by the present invention. The deployment and operation and maintenance intelligent body construction device based on the multimodal big model described below and the deployment and operation and maintenance intelligent body construction method based on the multimodal big model described above can be referenced to each other.
[0111] refer to Figure 5 , Figure 5 It is a structural diagram of the deployment and operation and maintenance intelligent body construction device based on the multimodal large model provided by the present invention, which includes: a construction module 501, a combination module 502, an acquisition module 503, a retrieval module 504 and a division module 505.
[0112] A construction module 501 is used to construct an RPA application based on an operating system handle and a multimodal large model, wherein the multimodal large model is used for image recognition and text recognition; A combination module 502 is used to perform permutations and combinations based on the atomic capabilities of the RPA applications to obtain multiple RPA tool chains, wherein each of the multiple RPA tool chains is used to perform different target tasks; An acquisition module 503 is used to acquire a preset search enhancement generation model and a preset standardized knowledge base document; The retrieval module 504 is used to input the standardized knowledge base document into the retrieval enhancement generation model to obtain the knowledge memory and retrieval module, and to construct the outer brain of the deployment operation and maintenance intelligent agent based on the long and short-term memory of the knowledge memory and retrieval module and the preset large language model, wherein the outer brain is used to recognize natural language input; The division module 505 is used to divide multiple RPA tool chains into multiple task executors according to target tasks, and to build an inner brain for deploying the operation and maintenance intelligent body based on multiple task executors, wherein the inner brain is used to call the inner task executor to execute the executable task flow corresponding to the natural language input.
[0113] Specifically, the above-mentioned deployment and operation and maintenance intelligent body construction device based on the multimodal big model provided by the present invention can implement all the method steps implemented by the above-mentioned deployment and operation and maintenance intelligent body construction method embodiment based on the multimodal big model, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0114] Figure 6 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute a method for constructing a deployment and operation and maintenance intelligent agent based on a multimodal large model, the method comprising: constructing an RPA application based on an operating system handle and a multimodal large model, wherein the multimodal large model is used for image recognition and text recognition; arranging and combining based on the atomic capabilities of the RPA application to obtain multiple RPA tool chains, wherein each of the multiple RPA tool chains is used to execute different target tasks; obtaining a preset retrieval enhancement generation model and a preset standardized knowledge base document; inputting the standardized knowledge base document into the retrieval enhancement generation model to obtain a knowledge memory and retrieval module, and constructing an outer brain of a deployment and operation and maintenance intelligent agent based on the long and short-term memory of the knowledge memory and retrieval module and the preset large language model, wherein the outer brain is used to recognize natural language input; dividing multiple RPA tool chains into multiple task executors according to target tasks, and constructing an inner brain of a deployment and operation and maintenance intelligent agent based on multiple task executors, wherein the inner brain is used to call the inner task executor to execute an executable task flow corresponding to the natural language input.
[0115] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0116] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the deployment and operation and maintenance intelligent body construction method based on the multimodal large model provided by the above methods, and the method includes: building an RPA application based on an operating system handle and a multimodal large model, wherein the multimodal large model is used for image recognition and text recognition; arranging and combining the atomic capabilities of the RPA application to obtain multiple RPA tool chains, wherein each of the multiple RPA tool chains is respectively used To perform different target tasks; obtain a preset retrieval enhancement generation model and a preset standardized knowledge base document; input the standardized knowledge base document into the retrieval enhancement generation model to obtain a knowledge memory and retrieval module, and build an outer brain for deploying and operating an intelligent body based on the knowledge memory and retrieval module and the long and short-term memory of a preset large language model, wherein the outer brain is used to recognize natural language input; divide multiple RPA tool chains into multiple task executors according to target tasks, and build an inner brain for deploying and operating an intelligent body based on multiple task executors, wherein the inner brain is used to call the inner task executor to execute the executable task flow corresponding to the natural language input.
[0117] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the deployment and operation and maintenance intelligent body construction method based on the multimodal big model provided by the above-mentioned methods, the method comprising: constructing an RPA application based on an operating system handle and a multimodal big model, wherein the multimodal big model is used for image recognition and text recognition; arranging and combining the atomic capabilities of the RPA application to obtain multiple RPA tool chains, wherein each RPA tool chain in the multiple RPA tool chains is used to execute different target tasks respectively; obtaining a preset The retrieval enhancement generation model and the preset standardized knowledge base document are set; the standardized knowledge base document is input into the retrieval enhancement generation model to obtain the knowledge memory and retrieval module, and based on the knowledge memory and retrieval module and the long and short-term memory of the preset large language model, the outer brain of the deployment operation and maintenance intelligent body is constructed, wherein the outer brain is used to recognize natural language input; multiple RPA tool chains are divided into multiple task executors according to the target tasks, and the inner brain of the deployment operation and maintenance intelligent body is constructed based on the multiple task executors, wherein the inner brain is used to call the inner task executor to execute the executable task flow corresponding to the natural language input.
[0118] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for constructing a deployment and operation agent based on a multimodal large model. It is characterized in that include: Building an RPA application based on an operating system handle and a multimodal large model, wherein the multimodal large model is used for image recognition and text recognition; Arrange and combine the atomic capabilities of the RPA application to obtain multiple RPA tool chains, wherein each of the multiple RPA tool chains is used to execute different target tasks; Obtaining a preset retrieval enhancement generation model and a preset standardized knowledge base document; Input the standardized knowledge base document into the retrieval enhancement generation model to obtain a knowledge memory and retrieval module, and construct an outer brain of the deployment operation and maintenance intelligent agent based on the long and short-term memory of the knowledge memory and retrieval module and the preset large language model, wherein the outer brain is used to recognize natural language input; The multiple RPA tool chains are divided into multiple task executors according to the target tasks, and the inner brain of the deployment and operation and maintenance intelligent body is constructed based on the multiple task executors, wherein the inner brain is used to call the inner task executors to execute the executable task flow corresponding to the natural language input.
2. According to the method for constructing a deployment and operation agent based on a multimodal large model according to claim 1, It is characterized in that The RPA application is constructed based on the operating system handle and the multimodal large model, including: Get the operating system handle and the preset multimodal large model; In the local environment, based on the operating system handle, determining basic capabilities of the RPA application, wherein the basic capabilities include at least one of the following: input simulation, process control, result output, and system interaction; In a remote environment, based on the multimodal big model, the composite capability of the RPA application is determined, and the composite capability includes at least one of the following: intent analysis and report analysis.
3. According to the method for constructing a deployment and operation agent based on a multimodal large model according to claim 1, It is characterized in that Before obtaining the preset search enhancement generation model and the preset standardized knowledge base document, the method further includes: Obtaining standard training data, wherein the standard training data includes standardized processes for deployment and operation, historical interactive feedback processes, and historical interactive feedback results; Vectorizing the standard training data to obtain a training data vector; Storing the training data vector in a vector database; A standardized knowledge base document is constructed based on the vector database.
4. According to the method for constructing a deployment and operation agent based on a multimodal large model according to claim 1, It is characterized in that The method further comprises: Obtaining the natural language input; The natural language input is retrieved and recognized through the outer brain of the deployment and operation intelligent agent to obtain the user intention; The user intention is split through the outer brain of the deployment operation and maintenance intelligent body to obtain at least one executable task flow and the execution order of the executable task flow.
5. According to the method for constructing a deployment and operation agent based on a multimodal large model according to claim 4, It is characterized in that After splitting the user intention to obtain at least one executable task flow and the execution order of the executable task flow, the method further includes: Determine, through the inner brain of the deployment and operation agent, a plurality of inner task executors based on the executable task flow and the execution order; Through the inner brain of the deployment and operation and maintenance intelligent body, the multiple inner task executors are called to execute the executable task flow in the execution order to obtain the interactive feedback process and interactive feedback results.
6. According to the method for constructing a deployment and operation agent based on a multimodal large model according to claim 5, It is characterized in that The method further comprises: When the interactive feedback process does not match the executable task flow, determining an abnormal result through the inner brain of the deployment and operation intelligent agent; Through the inner brain of the deployment and operation and maintenance intelligent body, the inner task executor is called to feed back the abnormal result to the outer brain of the deployment and operation and maintenance intelligent body; Through the outer brain of the deployed operation and maintenance intelligent agent, searching the standardized knowledge base document based on the abnormal results and long and short-term memory; When a scenario solution corresponding to the abnormal result is retrieved, the executable task flow is completed through the outer brain of the deployment and operation and maintenance intelligent body to obtain a supplementary task flow command; and the supplementary task flow command is sent to the inner task executor of the inner brain of the deployment and operation and maintenance intelligent body; When no scenario solution corresponding to the abnormal result is retrieved and the abnormal result hinders the subsequent execution process of the executable task flow, the abnormal result is fed back to the user through the outer brain of the deployment and operation agent; In response to obtaining a feedback instruction input by the user for the abnormal result; The feedback instruction is sent to the inner task executor of the inner brain of the deployment and maintenance intelligent body through the outer brain of the deployment and maintenance intelligent body, and the exception handling process corresponding to the abnormal result is stored in the retrieval enhancement generation model.
7. A deployment and operation and maintenance intelligent agent construction device based on a multimodal large model, It is characterized in that include: A construction module, used to build an RPA application based on an operating system handle and a multimodal large model, wherein the multimodal large model is used for image recognition and text recognition; A combination module, used for performing permutations and combinations based on the atomic capabilities of the RPA applications to obtain multiple RPA tool chains, wherein each of the multiple RPA tool chains is used to perform different target tasks; An acquisition module, used to acquire a preset retrieval enhancement generation model and a preset standardized knowledge base document; A retrieval module is used to input the standardized knowledge base document into the retrieval enhancement generation model to obtain a knowledge memory and retrieval module, and to construct an outer brain of the deployment and operation agent based on the long and short-term memory of the knowledge memory and retrieval module and the preset large language model, wherein the outer brain is used to recognize natural language input; A division module is used to divide the multiple RPA tool chains into multiple task executors according to the target tasks, and to build an inner brain of the deployment and operation and maintenance intelligent body based on the multiple task executors, wherein the inner brain is used to call the inner task executor to execute the executable task flow corresponding to the natural language input.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, it implements the method for constructing a deployment and operation agent based on a multimodal large model as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, it implements the method for constructing a deployment and operation intelligent agent based on a multimodal large model as described in any one of claims 1 to 6.
10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, it implements the method for constructing a deployment and operation intelligent agent based on a multimodal large model as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Business handling method and device, electronic equipment and storage medium
CN115375114A
Fault self-recovery method and system based on large language model
CN117909119A
Intelligent device control method, device and storage medium
CN118246477A
Equipment operation and maintenance method and device based on large language model, electronic equipment and medium
CN118469541A
System using method and system based on large model
CN118656058A
Cited By
Construction method, device and equipment of intelligent criminal investigation platform and storage medium
CN119357339A
Method, device, equipment and storage medium for constructing intelligent criminal investigation platform
CN119357339B
RPA command automatic generation system and method based on AI Agent
CN119474349A
Model construction method for GUI intelligent agent, program product and storage medium
CN119576470A
Method for enhancing RPA digital employee planning ability based on retrieval enhancement generation mode
CN119831017A
