Enterprise-level large model agent application system supporting multi-modal collaboration

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an enterprise-level large-model agent application system supporting multi-modal collaboration, and the system comprises a user interaction terminal, an agent engine server, a knowledge engine server, a plug-in integration center, and a distributed storage unit. The agent engine server is responsible for intention recognition and task arrangement of a multi-modal input signal, and dynamically loads a differential reasoning strategy based on an environment isolation mechanism. And the knowledge engine server constructs a cross-modal semantic anchor point, analyzes an unstructured document into a tetrad knowledge unit, and realizes accurate recall of images and texts by using a hybrid retrieval algorithm. And the plug-in integration center executes outbound replacement and inbound restoration of the sensitive data through the context-aware dynamic desensitization gateway. According to the method, through a multi-modal semantic association and closed-loop verification mechanism, the problems of low complex document retrieval precision and leakage of external calling data are solved, and the service processing capacity and safety of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an enterprise-level large-scale intelligent agent application system that supports multimodal collaboration. Background Technology

[0002] With the widespread application of large language model technology, intelligent agent systems based on the Retrieval Augmentation (RAG) architecture have become important tools for enterprises to handle knowledge-intensive tasks. In real-world enterprise applications, business documents are often presented in a mixed text and image format, containing numerous statistical charts, structural diagrams, and corresponding explanatory text. However, existing document parsing and retrieval technologies often separate text from image elements within a document, or perform only simple optical character recognition on the images. This approach severs the semantic connection between unstructured visual elements and their surrounding contextual text. Consequently, when users use natural language queries to inquire about chart trends or image details, the system struggles to accurately retrieve the corresponding visual information through text semantic matching, thus impacting the intelligent agent's ability to understand and reason about complex documents.

[0003] Furthermore, enterprise-level intelligent agents often need to invoke external general-purpose large-scale model capabilities or third-party service interfaces when performing complex tasks. In this process, ensuring the privacy and security of core enterprise data while utilizing advanced external capabilities becomes a pressing issue. Directly sending raw requests containing sensitive information to the external environment poses a risk of data leakage, while traditional rule-based hard filtering or masking often destroys the semantic integrity and referential relationships of the input data, causing external models to fail to output correct results due to a lack of contextual information. Simultaneously, existing intelligent agent system architectures typically lack differentiated resource scheduling mechanisms for different development stages, making it difficult to balance the observability requirements of end-to-end logs during development and debugging with the performance requirements of low latency and high throughput during production operation. Moreover, there is a lack of effective closed-loop verification methods for automatically generated business knowledge by the intelligent agent, easily leading to the accumulation of erroneous information in the knowledge base.

[0004] Therefore, this invention proposes an enterprise-level large-scale intelligent agent application system that supports multimodal collaboration to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an enterprise-level large-scale intelligent agent application system that supports multimodal collaboration. It solves the problems of low accuracy in complex document retrieval due to the lack of semantic association between text and images in existing technologies, as well as the difficulty in balancing data privacy protection and contextual semantic integrity during external service calls.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an enterprise-level large-scale intelligent agent application system supporting multimodal collaboration, comprising: The user interaction terminal is configured to collect multimodal input signals from users, including text commands, voice data streams, and unstructured document files, and to display the execution results fed back by the system. The intelligent agent engine server is configured with a large model interface component, a task orchestration component, and a context management component; the intelligent agent engine server is used to receive multimodal input signals, perform intent recognition and task decomposition on the multimodal input signals, and generate a thought chain path containing multiple execution nodes; The knowledge engine server is equipped with a multimodal parsing component and a vector retrieval engine for structuring and storing heterogeneous data within the enterprise. The plugin integration center is configured with a protocol conversion gateway for connecting to external service interfaces of the enterprise. The plugin integration center is used to receive function call requests from the intelligent agent engine server, perform external operations after filtering through security policies, and feed back the execution results to the intelligent agent engine server. Distributed storage units are used to store system configuration information, user session state data, and long-term memory data.

[0007] Preferably, the intelligent agent engine server is internally configured with an environment isolation management module; the environment isolation management module builds physically isolated or logically isolated running spaces at the underlying level based on containerized virtualization technology, the running spaces include at least a development running environment, an audit running environment and a production running environment, and assigns a unique environment identifier to different running spaces; The distributed storage unit is configured with a shared memory storage area that is independent of the operating space; The environment isolation management module is configured with a cross-environment synchronization protocol, which enables agent instances running in the development environment, audit environment and production environment to have read and update permissions to the shared memory storage area, provided they have valid access credentials.

[0008] Preferably, the intelligent agent engine server is also equipped with an inference strategy control module; The reasoning strategy control module is pre-set with a strategy mapping table that establishes a correspondence between environment identifiers and thought chain strategy functions, and dynamically loads the corresponding thought chain execution strategy according to the environment identifier of the environment where the current agent instance is located. When the environment identifier corresponds to the development and operation environment, a detailed inference strategy is loaded, and the agent is configured to record intermediate inference process data, including prompt word input vectors, original thought chain text, and token consumption statistics, when executing the node of the task orchestration process. When the environment identifier corresponds to the production and operation environment, the high-speed inference strategy is loaded, the intelligent agent is configured to execute the thought chain compression algorithm, the inference process log output of intermediate nodes is blocked, and the task planning results and final decision data are retained.

[0009] Preferably, the multimodal parsing component of the knowledge engine server is used to receive multi-source heterogeneous data streams and call the optical character recognition component, automatic speech recognition component or computer vision component to perform standardized content extraction on the data streams to obtain basic content data; During the parsing of a composite document containing multimedia elements, the knowledge engine server is configured to construct cross-modal semantic anchors. This construction process includes: Extract the logical coordinates of the multimedia element in the original document; extract the preceding or following text paragraphs that are immediately adjacent to the multimedia element in the document reading order as contextual semantic fragments; input the contextual semantic fragments into a pre-trained embedding model to generate a high-dimensional vector representation, i.e., a semantic anchor vector.

[0010] Preferably, the knowledge engine server encapsulates the parsed data into quadruple knowledge units containing semantic anchors and writes them into the database of the vector retrieval engine; the quadruple knowledge unit includes: A unique identifier in the global namespace; Basic content data, for chart elements, includes numerical text within the chart and the chart title; Semantic anchor vectors are used to establish index relationships between non-textual elements and textual semantics; Logical coordinates are used to restore the display position of the element in the original document during retrieval recall.

[0011] Preferably, the knowledge engine server is configured with a hybrid retrieval engine for executing a hybrid retrieval algorithm; the hybrid retrieval algorithm is configured to calculate a comprehensive similarity score between the query object and knowledge units in the knowledge base, the comprehensive similarity score being composed of two weighted parts: The first part is the cosine similarity of the vector space, which is used to measure the relevance between the vector representation of the query object and the semantic anchor vector. The second part is the score based on keyword matching, which is used to measure the degree of matching between the text content of the query object and the basic content data. The knowledge engine server sorts the knowledge units according to the comprehensive similarity score, and reconstructs a complete context fragment containing mixed text and image information based on the logical coordinates and returns it to the agent engine server.

[0012] Preferably, the plug-in integration center integrates a context-aware dynamic de-identification gateway; The context-aware dynamic desensitization gateway configuration is used to intercept outbound call requests initiated by the intelligent agent engine server, load the security policy configuration that defines the set of sensitive entity types, and perform named entity recognition or regular expression matching on the request data. The context-aware dynamic desensitization gateway generates a unique desensitization placeholder for each sensitive entity that belongs to the sensitive entity type set, maintains a local mapping table that records the correspondence between real entities and the desensitization placeholders, and sends the request data to an external system after replacing the sensitive entities in the request data with the corresponding desensitization placeholders.

[0013] Preferably, the context-aware dynamic de-identification gateway is further configured to intercept inbound response data from external systems and execute reverse restoration logic; the reverse restoration logic includes: The system iterates through the inbound response data, identifies the de-identified placeholders contained therein, finds the corresponding original real entity according to the local mapping table, replaces the de-identified placeholders back with the original real entity, and feeds back the restored data to the intelligent agent engine server.

[0014] Preferably, the knowledge engine server internally comprises a main knowledge base and a knowledge temporary storage area; the knowledge temporary storage area is used to receive and temporarily store business insight data, which is encapsulated as candidate knowledge units, generated by the intelligent agent engine server during task execution. The intelligent agent engine server is configured to instantiate an auditing intelligent agent to initiate adversarial verification of the candidate knowledge unit when new data is written to the knowledge temporary storage area. The auditing agent is configured to retrieve associated existing knowledge sets in the main knowledge base based on the candidate knowledge units, load business logic constraint sets, and execute verification functions to calculate the confidence score of the candidate knowledge units.

[0015] Preferably, the knowledge engine server is configured with an evolution controller, used to execute a dynamic evolution strategy based on the confidence score; the evolution controller compares the confidence score with a preset merging threshold: If the confidence score is greater than or equal to the merging threshold, a merging operation is performed, the candidate knowledge unit is transferred from the knowledge temporary storage area to the main knowledge base, and the vector index is updated; If the confidence score is less than the merging threshold, the merging operation is rejected, and the candidate knowledge unit is marked as abnormal data, generating an anomaly log containing the reason for the verification failure.

[0016] This invention provides an enterprise-level large-scale intelligent agent application system that supports multimodal collaboration. It has the following beneficial effects: 1. By constructing a cross-modal semantic anchor and quadruple knowledge storage structure, this invention establishes a strong logical association between non-textual elements (such as statistical charts and illustrations) and their contextual text semantics. This mechanism enables the system to accurately recall visually relevant chart data based on semantic similarity when receiving only natural language queries, without relying on visual feature matching of the images themselves, effectively solving the problem of fragmented image-text relationships in traditional retrieval schemes. Simultaneously, by reconstructing context fragments using logical coordinates, it ensures that the recalled knowledge retains the original document's layout and structural information, thereby improving the inference accuracy of the agent when handling complex document analysis tasks.

[0017] 2. This invention implements a full lifecycle management mechanism based on environment isolation and differentiated inference strategies. The system dynamically loads execution strategies according to the identifiers of the current operating environment. In the development environment, it retains a full set of inference logs to support logic backtracking and debugging, while in the production environment, it employs a high-speed compression strategy to reduce system latency and resource consumption. This hierarchical control not only resolves the contradiction between the convenience of model debugging and online operating efficiency, but also ensures the continuity and consistency of user experience throughout the system iteration process through a cross-environment shared memory synchronization mechanism.

[0018] 3. By integrating a context-aware dynamic de-identification gateway, this invention achieves secure interconnection between enterprise intranet intelligent agents and external third-party services. This gateway identifies sensitive entities and replaces them with semantically preserved placeholders when making outbound requests, and performs reverse restoration when responding to inbound requests. This blocks the leakage path of sensitive data to external platforms without disrupting the contextual logic required for model inference. This transparent de-identification mechanism at the middleware layer enables enterprises to securely leverage the capabilities of external general-purpose models to expand their business while meeting stringent data privacy compliance requirements. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall architecture of the present invention; Figure 2 This is a schematic diagram of the method flow of the present invention; Figure 3 This is a schematic diagram illustrating the principle of multimodal parsing and hybrid retrieval based on semantic anchors in this invention; Figure 4 This is a schematic diagram illustrating the dynamic privacy desensitization and knowledge closed-loop evolution logic of the present invention.

[0020] Among them, 101 is the user interaction terminal; 102 is the intelligent agent engine server; 103 is the knowledge engine server; 104 is the plug-in integration center; and 105 is the distributed storage unit. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 , Figure 1 This is a schematic diagram of the overall architecture of an enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to an embodiment of the present invention. The enterprise-level large-scale intelligent agent application system supporting multimodal collaboration provided by the present invention includes: The aforementioned components are deployed in an enterprise private cloud or intranet server cluster and connect to each other via an encrypted communication protocol. The user interaction terminal 101 is configured to receive multimodal input signals from users, including text commands, voice data streams, and unstructured document files, and to display the execution results fed back by the system.

[0023] The intelligent agent engine server 102 is the core control unit of the system, equipped with a large model interface component, a task orchestration component, and a context management component. The intelligent agent engine server 102 receives multimodal input signals from the user interaction terminal 101, performs intent recognition and task decomposition on the input signals, and generates a thought chain path containing multiple execution nodes. The intelligent agent engine server 102 establishes bidirectional data connections with the knowledge engine server 103 and the plug-in integration center 104 respectively, and schedules corresponding downstream resources according to the node type of the thought chain path.

[0024] The knowledge engine server 103 is equipped with a multimodal parsing component and a vector retrieval engine for structuring and storing heterogeneous data within the enterprise. The knowledge engine server 103 not only responds to the read request of the intelligent agent engine server 102 to return knowledge slices, but also has a write interface to receive business insight data generated by the intelligent agent engine server 102, realizing bidirectional data flow.

[0025] The plug-in integration center 104 is equipped with a protocol conversion gateway for connecting the enterprise resource planning system, office automation system and external third-party service interfaces. The plug-in integration center 104 receives function call requests from the intelligent agent engine server 102, performs external operations after filtering through security policies, and feeds back the execution results to the intelligent agent engine server 102.

[0026] The distributed storage unit 105 includes a relational database instance and a key-value pair storage instance, which are used to store system configuration information, user session state data and long-term memory data. The intelligent agent engine server 102 and the knowledge engine server 103 are both connected to the distributed storage unit 105 to realize the persistence and sharing of state data.

[0027] See attached document Figure 2 , Figure 2 This is a flowchart illustrating a method for operating an enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to an embodiment of the present invention. The present invention provides a method for operating an enterprise-level large-scale intelligent agent application system supporting multimodal collaboration, comprising the following steps: S100, the user interaction terminal 101 collects the user's multimodal input signals and sends the multimodal input signals to the intelligent agent engine server 102; S200, the intelligent agent engine server 102 uses the environment isolation management module to identify the environment identifier of the current operating environment, and uses the inference strategy control module to load the corresponding thought chain execution strategy according to the environment identifier. S300, the intelligent agent engine server 102 executes the loaded thought chain execution strategy to perform intent recognition and task path decomposition on multimodal input signals; S400, if the task path involves querying internal enterprise data, the intelligent agent engine server 102 sends a retrieval instruction to the knowledge engine server 103 and obtains the structured knowledge data returned by the knowledge engine server 103. S500, if the task path involves the execution of business functions, the intelligent agent engine server 102 sends a call instruction to the plug-in integration center 104 and obtains the execution result data returned by the plug-in integration center 104. S600, the intelligent agent engine server 102 generates response feedback based on structured knowledge data or execution result data, and sends the response feedback to the user interaction terminal 101.

[0028] The technical details of each of the above process steps will be explained in detail below, with specific implementation steps as an example.

[0029] See attached document Figure 2 It illustrates the logical structure of intelligent agent operation environment construction and inference control according to an embodiment of the present invention; the intelligent agent engine server 102 is internally configured with an environment isolation management module and an inference strategy control module, which work together to realize hierarchical management and execution optimization of the entire life cycle of the intelligent agent; the distributed storage unit 105 is configured with a shared memory storage area to maintain cognitive consistency across environments.

[0030] The implementation methods for constructing the intelligent agent's operating environment and controlling inference are described in detail below, using specific process steps S201 to S203 as examples: S201, the environment isolation management module, builds physically or logically isolated running spaces at the underlying layer of the server cluster based on containerized virtualization technology.

[0031] The environment isolation management module is equipped with an environment identifier generator, which is used to assign unique environment identifiers to different operating spaces. The runtime environment includes, but is not limited to, development runtime environment, audit runtime environment, and production runtime environment, each corresponding to a set of environment identifiers. The development and runtime environment is configured with open debugging ports and full log read / write permissions for building agent workflows and unit testing; the audit runtime environment is configured with a preset set of security rules for scanning for logical vulnerabilities and verifying consistency of agent versions; the production runtime environment is configured with a high-concurrency load balancing strategy for providing business services to end users; the specific implementation of containerization technology, such as namespace isolation or control group resource restrictions, is well known to those skilled in the art and will not be elaborated here.

[0032] The shared memory storage area in the distributed storage unit 105 is configured as a persistent data layer independent of the aforementioned operating environment; the shared memory storage area uses a key-value pair or vector index structure to store the agent's long-term memory data. The environment isolation management module is configured with a cross-environment synchronization protocol, which enables agent instances running in development, audit, and production environments to have read and update permissions to the shared memory storage area, provided they have valid access credentials. This mechanism ensures that when an agent updates user preferences in the development environment, the update can be synchronized to the production environment in real time, maintaining the continuity of the user interaction experience across different version iterations.

[0033] S202, the inference strategy control module dynamically loads the corresponding thought chain execution strategy based on the environment identifier of the current agent instance's environment.

[0034] The inference policy control module has a pre-set policy mapping table, which establishes environment identifiers. With the mind chain strategy function The correspondence between them; when the intelligent agent engine server 102 receives an execution request, the inference strategy control module intercepts the request, extracts the identifier of the current running environment, and executes the following judgment logic: If the environment identifier is Load detailed inference strategies; configure the agent to fully record intermediate inference process data at each node of the task orchestration process; intermediate inference process data includes prompt word input vectors, original thought chain text generated by the large model, input and output parameters of each step, and token consumption statistics.

[0035] If the environment identifier is The system loads a rapid reasoning strategy; the rapid reasoning strategy configures the agent to execute a thought chain compression algorithm. This algorithm, by adjusting system prompts or post-processing filters, blocks the output of the reasoning process logs of intermediate nodes, retaining only the task planning results and the final decision data.

[0036] For any intelligent agent task The inference strategy control module controls the generation of intermediate state logs. Follow the logic defined by the following formula: ; In the formula, Indicates the current environment identifier; Indicates detailed mode; Indicates compact mode; This represents a log set that includes all intermediate inference steps and token consumption statistics. This represents a log set that, after pruning, contains only key decision nodes.

[0037] S203, the intelligent agent engine server 102 executes the loaded strategy and synchronously updates the shared memory; When performing specific tasks, the intelligent agent engine server 102 calls the large model component to generate inference results; if the current environment is a development / running environment, the system will... Real-time push notifications are sent to the front-end debugging panel to assist developers in single-step breakpoint debugging and logic backtracking; if the current environment is production, the system will only push notifications to the front-end debugging panel. Write to the runtime log to reduce storage overhead and system response latency; after the task is completed, regardless of the inference strategy used, the agent engine server 102 will write the generated state change data to the shared memory storage area. The update operation is represented as follows: ; In the formula, Represents intelligent agents The current state of cognition; This represents long-term memory data stored in the shared memory storage area; This step represents the unique identifier of the associated user; it ensures that differences in inference strategies under different environments only affect the processing of process data, without affecting the consistency of the state of core business data.

[0038] See attached document Figure 3 The knowledge engine server 103 is configured with a multimodal parsing component and a vector database interface to perform standardization processing and structured mapping of unstructured data. The implementation methods for multimodal data parsing and semantic anchor construction are described in detail below, using specific process steps S301 to S304: S301, the knowledge engine server 103 receives multi-source heterogeneous data streams uploaded from the user interaction terminal 101 or automatically collected by the system.

[0039] Multi-source heterogeneous data streams include, but are not limited to, portable document format (PDF) files, office software documents (Word / Excel), image files (JPG / PNG), and audio stream data; the knowledge engine server 103 is configured with a file type recognizer, which determines the data type based on the file header information or extension, and routes the data stream to the corresponding parsing queue; for plain text data or simple key-value pair data, the system directly performs word segmentation and vectorization processing; for composite documents containing images, charts, or audio, the system triggers a parsing agent to perform deep content extraction.

[0040] S302, the knowledge engine server 103 calls the optical character recognition (OCR) component, automatic speech recognition (ASR) component, or computer vision (CV) component to extract standardized content from the routed data.

[0041] The parsing agent performs hierarchical scanning of the input unstructured data. When processing image files or embedded images in documents, it calls the OCR component to extract text information and the CV component to generate image captions. When processing audio data, it calls the ASR component to convert it into a text transcript. The specific algorithmic implementations of the OCR, ASR, and CV components, such as feature extraction based on convolutional neural networks or sequence transcription based on Transformers, are well-known technologies in the field and will not be elaborated upon here. The system defines the extracted text information as basic content data.

[0042] S303, the knowledge engine server 103 constructs cross-modal semantic anchors during the parsing of compound documents to establish logical connections between multimedia elements and text context.

[0043] For complex documents containing charts, images, and text, the system does not separate the extracted basic content data from the original document structure; while extracting multimedia elements, the parsing agent executes semantic anchor extraction logic; this logic includes extracting the logical coordinates of the multimedia elements and extracting the contextual semantic fragments of the multimedia elements.

[0044] Logical coordinates represent the topological or spatial position of a multimedia element within the original document. For streaming documents (such as HTML or Word), logical coordinates are the node path of the element in the Document Object Model (DOM) tree; for layout documents (such as PDF), logical coordinates are the normalized position parameters of the element in the page coordinate system. ,in, The coordinates of the top left corner of the element. This represents the width and height of the element.

[0045] The contextual semantic fragment is the preceding or following text paragraph that is immediately adjacent to the multimedia element in the document reading order. The system inputs this contextual semantic fragment into a pre-trained embedding model to generate a high-dimensional vector representation, namely a semantic anchor vector.

[0046] S304, the knowledge engine server 103 encapsulates the parsed data into four-tuple knowledge units containing semantic anchors and writes them into the vector database; the system defines the knowledge units. The storage structure is shown in the following formula: ; In the formula: This represents the unique identifier of the knowledge unit in the global namespace; This represents the basic content data extracted in step S302. For chart elements, this data includes the numerical text within the chart and the chart title. This represents the set of logical coordinates extracted in step S303, which is used to restore the display position of the element in the original document during subsequent retrieval. The semantic anchor vector is calculated according to the following mapping relationship: ; In the formula, This represents a text embedding function. This refers to the contextual text fragment that is physically adjacent to the knowledge unit in the original document.

[0047] Through the aforementioned storage structure, the knowledge engine server 103 establishes an index relationship between non-textual elements (images / charts) and textual semantics at the database level. When subsequent agents perform retrieval based on natural language, the system can calculate the query vector and... By using similarity, visually relevant chart data can be recalled without relying on visual feature matching of the images themselves, thus solving the problem of lost image-text relationships in traditional retrieval schemes.

[0048] See attached document Figure 3The intelligent agent engine server 102 and the knowledge engine server 103 are connected through an internal high-speed network to collaboratively complete the parsing of user intent, task decomposition, and accurate retrieval of multimodal knowledge. The following describes in detail the implementation method of intelligent agent collaborative orchestration and semantic anchor-based hybrid retrieval, using specific process steps S401 to S404: S401, the intelligent agent engine server 102 receives user input signals and uses the intent recognition model to make task routing decisions.

[0049] The intelligent agent engine server 102 is configured with a pre-trained intent classifier; when it receives a multimodal input signal... At that time, the intent classifier calculates whether the input signal belongs to a preset task type set. Posterior probabilities of various types; set of task types This includes knowledge-based question-and-answer tasks, data analysis tasks, and external tool invocation tasks; the system selects the type with the highest probability as the dominant intent of the current session; if the dominant intent is determined to be a complex task (such as "analyzing the profit trend in this quarter's financial report"), the system triggers a multi-agent collaborative orchestration process; if the dominant intent is determined to be a single knowledge query, the retrieval process is triggered directly.

[0050] S402, the intelligent agent engine server 102, decomposes complex tasks into sub-task sequences based on a predefined workflow graph and distributes them to sub-intelligent agents with different functions.

[0051] The system maintains a workflow graph internally. ,in, Represents a set of functional nodes. Indicates the data flow direction edge; a set of functional nodes. This includes, but is not limited to, parsing intelligent agents, analyzing intelligent agents, and generating intelligent agents; the intelligent agent engine server 102 acts as a scheduler, activating specific paths in the graph based on the identified intent.

[0052] For example, when processing document analysis tasks that include charts, the scheduler first activates the parsing agent to perform structuring processing on the document and output intermediate structured data; then it activates the analysis agent to read the intermediate structured data and perform numerical calculations; finally, it activates the generating agent to convert the calculation results into natural language descriptions; the sub-agents pass context data to each other through shared memory or message queues to ensure the continuity of task execution.

[0053] S403, during the execution of a task, the intelligent agent engine server 102 initiates a hybrid retrieval request based on semantic anchors to the knowledge engine server 103.

[0054] When any sub-agent needs to obtain external knowledge support, a query object is generated. ;. Query object Includes natural language query text and query vectors generated through the embedded model The intelligent agent engine server 102 will query the object. Send to Knowledge Engine Server 103.

[0055] S404, the knowledge engine server 103 executes a hybrid retrieval algorithm to calculate the comprehensive similarity score between the query object and the knowledge units in the knowledge base, and returns the sorted knowledge slices.

[0056] The knowledge engine server 103 is configured with a hybrid retrieval engine that simultaneously performs vector similarity calculation and keyword matching calculation. To accurately retrieve unstructured elements (such as illustrations and statistical charts) that are strongly related to the text context, the system utilizes the semantic anchor vectors constructed in the aforementioned embodiments. Participate in the calculation.

[0057] For any knowledge unit in the knowledge base The system calculates its relationship with the query object based on the following formula. Overall similarity score : ; In the formula: The cosine similarity function for vector spaces is calculated using the following formula: This item measures the relevance of a user’s query intent to the semantic context (i.e., semantic anchors) surrounding a multimedia element. Even if a multimedia element itself (such as a landscape image without text) cannot be directly matched with a text query, the system can still recall it by matching its semantic anchors. This represents the keyword matching score based on the BM25 algorithm; this item measures the degree of matching between the query text and the explicit textual information contained within the knowledge unit (such as text in charts and image titles recognized by OCR); and The preset weighting coefficients are used, and they satisfy the following conditions: The system dynamically adjusts the weights based on the task type, for example, increasing them in tasks that emphasize semantic understanding. The value of .

[0058] The knowledge engine server-side scores based on comprehensive similarity. The knowledge units are sorted in descending order, and the top-K knowledge units with the highest scores are selected. Then, the system reads the logical coordinates from these knowledge units. The recalled content is reordered according to its physical order in the original document, and a complete context fragment containing mixed text and image information is reconstructed and returned to the agent engine server 102. This process ensures that the information obtained by the agent is not only accurate in content, but also retains the logical structure of the original document, thereby improving the accuracy of subsequent reasoning generation.

[0059] See attached document Figure 4 The plug-in integration center 104 integrates a protocol adapter and a dynamic de-identification gateway component, which work together to achieve secure interconnection between the enterprise intranet intelligent agent and external third-party services. The following section elaborates on the protocol adaptation and context-aware dynamic de-identification mechanism of the plug-in integration center 104, using specific process steps S501 to S504 as examples: S501, the plugin integration center 104 parses the interface specification files of external services and registers standardized calling interfaces.

[0060] The plugin integration center 104 is equipped with an interface parsing engine that supports reading OpenAPI specifications (Swagger), gRPC definition files, or custom JSON-Schema description files. When the administrator or system imports the aforementioned interface specification files, the parsing engine extracts the service address, path parameters, request body structure, and authentication method information, and generates corresponding virtual function stubs within the system. The intelligent agent engine server 102 can initiate requests to external services by calling these virtual function stubs without needing to concern itself with the underlying HTTP or TCP communication details. The parsing of interface specification files and the generation of client code are well-known technologies in the field of art and will not be described in detail here.

[0061] S502, when the plugin integration center 104 receives an outbound call request, it triggers context-aware dynamic desensitization logic to replace sensitive entities in the request data.

[0062] When the intelligent agent engine server 102 initiates an external call request, the dynamic desensitization gateway component in the plugin integration center 104 intercepts the outbound request; the gateway component loads the enterprise's preset security policy configuration, which defines the set of sensitive entity types. This includes, but is not limited to, names, ID numbers, bank account numbers, and trade secret keywords.

[0063] The gateway component processes the request text. Perform Named Entity Recognition (NER) or regular expression matching to identify entities belonging to... Specific sensitive entities To ensure that the external model can understand the contextual logic (e.g., distinguishing between "Zhang San" and "Li Si" as two different people), the gateway component generates a unique de-identified placeholder for each sensitive entity. Desensitization placeholder Tokens using semantically preserved formats (e.g.)<PERSON_1> (or hash value)

[0064] The gateway component maintains a local mapping table throughout the lifecycle of the current session. This is used to record the correspondence between real entities and placeholders; the outbound desensitization processing logic follows the following formula: ; Simultaneously update the mapping table: ; In the formula, This indicates the request text that has been anonymized and is ready to be sent to an external system. This represents a text replacement operation function; Represents the original sensitive entity data; This represents the generated temporary desensitization placeholder.

[0065] S503, the plug-in integration center 104 sends the de-identified request to the external system and receives the response data from the external system.

[0066] The plug-in integration center 104 uses the secure transport layer protocol to... Send to the target external service interface (such as a public cloud big model API or a third-party SaaS application); the external system is based on placeholders. The context is used to perform business logic processing or reasoning, and generate response text. Throughout the entire external processing, the external system can only obtain data that has no specific business meaning. Unable to obtain the company's true sensitive data This achieves privacy protection for data leaving the network.

[0067] S504, the plugin integration center 104 intercepts the inbound response data, executes the reverse restoration logic, and feeds back the restored data to the intelligent agent engine server 102.

[0068] The dynamic desensitization gateway component receives an external response. Afterwards, instead of directly forwarding it to the agent, an inbound restoration operation is performed; the gateway component traverses the response text to identify the de-identified placeholders it contains. And based on the local mapping table maintained in step S502 Find the corresponding original entity .

[0069] The inbound restoration processing logic follows the formula below: ; In the formula, This represents the final response text after restoration; all placeholders in this text have been replaced with the original business entity names; Plugin Integration Center 104 will... The response is returned to the agent engine server 102. For the agent, both the requests it sends and the responses it receives contain real data. The desensitization and restoration process is completed transparently in the middleware layer, ensuring the consistency of the agent's internal logic and the security of external calls.

[0070] See attached document Figure 4 The intelligent agent engine server 102 and the knowledge engine server 103 not only have a one-way retrieval call, but also a reverse knowledge writing and verification link, forming a closed-loop control system. The following section elaborates on the knowledge generation temporary storage mechanism and adversarial verification evolution strategy in conjunction with specific process steps S601 to S604: S601, the intelligent agent engine server 102 generates business insight data during the execution of tasks, and encapsulates the data into candidate knowledge units and sends them to the knowledge temporary storage area of ​​the knowledge engine server 103. The agent engine server 102 is configured with an output monitoring component. When the agent performs analysis or summary tasks and generates the final output, the output monitoring component extracts the structured information from the result and constructs candidate knowledge units. Candidate knowledge units It includes the knowledge content payload, the source agent identifier that generated the knowledge, the generation timestamp, and the associated task context vector.

[0071] The knowledge engine server 103 internally includes a main knowledge base. With knowledge temporary storage area Main knowledge base Used to provide retrieval services to all intelligent agents and store verified, trustworthy data; knowledge storage area. Used for temporary storage of candidate knowledge to be verified; when the knowledge engine server 103 receives... When a write request is received, write it to At this point, the data is not visible to regular search requests to prevent unverified data from contaminating the system's perception.

[0072] S602, the intelligent agent engine server 102 instantiates the auditing intelligent agent and initiates adversarial verification on the candidate knowledge units in the knowledge temporary storage area.

[0073] The system is configured with an asynchronous event listener, when When new data is written, a verification event is triggered; the agent engine server 102 responds to the event, schedules and instantiates one or more audit agents; the audit agent loads a preset adversarial verification prompt strategy, which requires the agent to try to find logical loopholes or factual conflicts in the candidate knowledge.

[0074] The review agent reads candidate knowledge units from the knowledge engine server 103. and based on The vector representation in the main knowledge base Retrieve related existing knowledge sets Simultaneously, the auditing agent loads the enterprise's pre-defined set of business logic constraints. ; S603, the auditing agent executes the verification function to calculate the confidence score of the candidate knowledge unit.

[0075] The auditing agent uses a Natural Language Inference (NLI) model or rule matching engine to execute the verification function. The verification process includes consistency testing and rule compliance testing; consistency testing judgment. Whether or not Semantic implications or contradictory relationships exist; rule compliance detection and judgment. Does it violate Hard constraints (such as data format constraints and numerical range constraints).

[0076] System-defined confidence score The calculation formula is as follows: ; In the formula: The consistency score is represented by calculation. The probability of obtaining information that does not conflict with existing knowledge; The score represents the rule compliance score, obtained through Boolean logic checks. These are preset weighting coefficients used to balance the importance of historical data consistency with business rule constraints.

[0077] S604, the knowledge engine server 103 executes a dynamic evolution strategy based on the confidence score to determine whether to merge or discard candidate knowledge.

[0078] The intelligent agent engine server 102 will calculate the... Feedback is sent to knowledge engine server 103; knowledge engine server 103 is configured with an evolution controller, which stores the merging threshold. Evolutionary Controller Comparison and The size of the main knowledge base is determined, and the state of the main knowledge base is updated based on the comparison results; the main knowledge base is... state of time Determined by the following piecewise function: ; when At that time, the system performs a merge operation, which will... From the knowledge temporary storage area Transfer to the main knowledge base And update the vector index, thereby completing the self-evolution of system cognition.

[0079] when At that time, the system refused the merge operation and will Marked as anomalous data; Knowledge Engine server-side 103 generated data containing Exception logs containing content and reasons for verification failure The log will then be pushed to the administrator's review queue; for logs involving numerical errors or logical paradoxes... The system directly performs a discard operation to ensure that the data quality of the main knowledge base is within a controllable range.

[0080] User interaction terminal 101, as the system's front-end contact point, is configured as a client application or web browser instance running on a general-purpose computing device, responsible for the acquisition of multimodal data and the visualization of execution results. The implementation methods of the user interaction terminal and hardware entities are described in detail below, using specific process steps S701 to S703: S701, the user interaction terminal 101 is equipped with a multimodal input interface for collecting user command signals and performing preprocessing.

[0081] The user interaction terminal 101 establishes a multi-channel input stream by calling the hardware driver interface of the host device. For text input, the terminal provides a code editor or natural language input box, supporting real-time rendering of Markdown syntax. For voice input, the terminal calls the microphone array to collect the audio stream, performs noise reduction and echo cancellation processing, and performs pulse code modulation (PCM) encoding according to a preset sampling rate (such as 16kHz). For image or document input, the terminal calls the camera or file system interface to convert the file into a binary byte stream (Base64 or Multipart format).

[0082] The user interaction terminal 101 is configured with an input buffer to encapsulate the data of the different modalities into standardized request messages. The request message includes a session identifier, input data payload, and current timestamp. The terminal sends the request message to the intelligent agent engine server 102 through an encrypted transmission channel (such as HTTPS / WebSocket).

[0083] S702, the user interaction terminal 101 establishes a long connection with the intelligent agent engine server 102 and performs adaptive rendering based on the received streaming data.

[0084] To support streaming responses generated from large models, the user interaction terminal 101 uses Server Send Event (SSE) or WebSocket protocol to maintain bidirectional communication; the terminal internally maintains a rendering state machine that switches the display mode according to the type of data packet received.

[0085] When the intelligent agent engine server 102 is in the development and runtime environment, the terminal receives a data stream containing intermediate inference logs; the rendering state machine activates the split-screen display mode, rendering the final business response in the left window, and displaying the intelligent agent's thought chain steps, tool call parameters, and token consumption statistics in the right debugging window in real time to assist users in logic backtracking.

[0086] When the intelligent agent engine server 102 is in a production environment, the terminal only receives the pruned final result data stream; the rendering state machine activates a minimalist display mode, shielding all intermediate technical details and displaying only structured answers, charts, or file download links through rich text components.

[0087] S703, this application embodiment also provides an electronic device for running the various functional modules of the user interaction terminal 101 or the server described above.

[0088] The electronic device includes a memory, a processor, and a communication bus; the memory is used to store computer-executable instructions, and the processor is connected to the memory via the communication bus; when the processor executes the computer-executable instructions in the memory, it implements the enterprise-level large-scale intelligent agent application system operation method supporting multimodal collaboration as described in any of the above embodiments.

[0089] The memory can be volatile or non-volatile, including but not limited to random access memory (RAM), read-only memory (ROM), flash memory, or solid-state drive (SSD). The processor can be a general-purpose central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA), configured to perform a large number of vector matrix operations and logic control tasks. The communication bus is configured as an Industry Standard Architecture (ISA) bus, Peripheral Component Interconnect (PCI) bus, or Extended Industry Standard Architecture (EISA) bus, used to transmit data signals between the processor, memory, and network interface.

[0090] Furthermore, this application also provides a computer-readable storage medium storing a computer program thereon; when the computer program is executed by a processor, it implements the steps of the method described in any of the above embodiments; the computer-readable storage medium includes, but is not limited to, a USB flash drive, a portable hard drive, an optical disc, a magnetic disk, or cloud storage space provided by a cloud computing platform.

[0091] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An enterprise-level large-scale intelligent agent application system supporting multimodal collaboration, characterized in that, include: The user interaction terminal is configured to collect multimodal input signals from users, including text commands, voice data streams, and unstructured document files, and to display the execution results fed back by the system. The intelligent agent engine server is configured with a large model interface component, a task orchestration component, and a context management component; the intelligent agent engine server is used to receive multimodal input signals, perform intent recognition and task decomposition on the multimodal input signals, and generate a thought chain path containing multiple execution nodes; The knowledge engine server is equipped with a multimodal parsing component and a vector retrieval engine for structuring and storing heterogeneous data within the enterprise. The plugin integration center is configured with a protocol conversion gateway for connecting to external service interfaces of the enterprise. The plugin integration center is used to receive function call requests from the intelligent agent engine server, perform external operations after filtering through security policies, and feed back the execution results to the intelligent agent engine server. Distributed storage units are used to store system configuration information, user session state data, and long-term memory data.

2. The enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to claim 1, characterized in that, The intelligent agent engine server is internally configured with an environment isolation management module; the environment isolation management module builds physically or logically isolated running spaces at the bottom layer based on containerized virtualization technology, the running spaces include at least a development running environment, an audit running environment and a production running environment, and assigns a unique environment identifier to different running spaces; The distributed storage unit is configured with a shared memory storage area that is independent of the operating space; The environment isolation management module is configured with a cross-environment synchronization protocol, which enables agent instances running in the development environment, audit environment and production environment to have read and update permissions to the shared memory storage area, provided they have valid access credentials.

3. The enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to claim 2, characterized in that, The intelligent agent engine server is also equipped with an inference strategy control module; The reasoning strategy control module is pre-set with a strategy mapping table that establishes a correspondence between environment identifiers and thought chain strategy functions, and dynamically loads the corresponding thought chain execution strategy according to the environment identifier of the environment where the current agent instance is located. When the environment identifier corresponds to the development and operation environment, a detailed inference strategy is loaded, and the agent is configured to record intermediate inference process data, including prompt word input vectors, original thought chain text, and token consumption statistics, when executing the node of the task orchestration process. When the environment identifier corresponds to the production and operation environment, the high-speed inference strategy is loaded, the intelligent agent is configured to execute the thought chain compression algorithm, the inference process log output of intermediate nodes is blocked, and the task planning results and final decision data are retained.

4. The enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to claim 1, characterized in that, The multimodal parsing component of the knowledge engine server is used to receive multi-source heterogeneous data streams and call optical character recognition component, automatic speech recognition component or computer vision component to perform standardized content extraction on the data streams to obtain basic content data; During the parsing of a composite document containing multimedia elements, the knowledge engine server is configured to construct cross-modal semantic anchors. This construction process includes: Extract the logical coordinates of the multimedia elements in the original document; Extract the preceding or following text paragraphs that are immediately adjacent to the multimedia element in the document reading order as contextual semantic fragments; input the contextual semantic fragments into a pre-trained embedding model to generate a high-dimensional vector representation, i.e., a semantic anchor vector.

5. The enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to claim 1, characterized in that, The knowledge engine server encapsulates the parsed data into four-tuple knowledge units containing semantic anchors and writes them into the database of the vector retrieval engine; The quadruple knowledge unit includes: A unique identifier in the global namespace; Basic content data, for chart elements, includes numerical text within the chart and the chart title; Semantic anchor vectors are used to establish index relationships between non-textual elements and textual semantics; Logical coordinates are used to restore the display position of the element in the original document during retrieval recall.

6. The enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to claim 5, characterized in that, The knowledge engine server is configured with a hybrid retrieval engine to execute a hybrid retrieval algorithm. The hybrid retrieval algorithm is configured to calculate a comprehensive similarity score between the query object and knowledge units in the knowledge base. This comprehensive similarity score is composed of two weighted components: The first part is the cosine similarity of the vector space, which is used to measure the relevance between the vector representation of the query object and the semantic anchor vector. The second part is the score based on keyword matching, which is used to measure the degree of matching between the text content of the query object and the basic content data. The knowledge engine server sorts the knowledge units according to the comprehensive similarity score, and reconstructs a complete context fragment containing mixed text and image information based on the logical coordinates and returns it to the agent engine server.

7. The enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to claim 1, characterized in that, The plugin integration center integrates a context-aware dynamic desensitization gateway. The context-aware dynamic desensitization gateway configuration is used to intercept outbound call requests initiated by the intelligent agent engine server, load the security policy configuration that defines the set of sensitive entity types, and perform named entity recognition or regular expression matching on the request data. The context-aware dynamic desensitization gateway generates a unique desensitization placeholder for each sensitive entity that belongs to the sensitive entity type set, maintains a local mapping table that records the correspondence between real entities and the desensitization placeholders, and sends the request data to an external system after replacing the sensitive entities in the request data with the corresponding desensitization placeholders.

8. An enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to claim 7, characterized in that, The context-aware dynamic desensitization gateway is also configured to intercept inbound response data from external systems and execute reverse restoration logic; The reverse restoration logic includes: The system iterates through the inbound response data, identifies the de-identified placeholders contained therein, finds the corresponding original real entity according to the local mapping table, replaces the de-identified placeholders back with the original real entity, and feeds back the restored data to the intelligent agent engine server.

9. An enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to claim 1, characterized in that, The knowledge engine server is internally divided into a main knowledge base and a knowledge temporary storage area; the knowledge temporary storage area is used to receive and temporarily store business insight data generated by the intelligent agent engine server during task execution and packaged into candidate knowledge units. The intelligent agent engine server is configured to instantiate an auditing intelligent agent to initiate adversarial verification of the candidate knowledge unit when new data is written to the knowledge temporary storage area. The auditing agent is configured to retrieve associated existing knowledge sets in the main knowledge base based on the candidate knowledge units, load business logic constraint sets, and execute verification functions to calculate the confidence score of the candidate knowledge units.

10. An enterprise-level large-scale intelligent agent application system supporting multimodal collaboration according to claim 9, characterized in that, The knowledge engine server is configured with an evolution controller, which executes a dynamic evolution strategy based on the confidence score; the evolution controller compares the confidence score with a preset merging threshold: If the confidence score is greater than or equal to the merging threshold, a merging operation is performed, the candidate knowledge unit is transferred from the knowledge temporary storage area to the main knowledge base, and the vector index is updated; If the confidence score is less than the merging threshold, the merging operation is rejected, and the candidate knowledge unit is marked as abnormal data, generating an anomaly log containing the reason for the verification failure.

Images