Multi-modal Agent coordinated reasoning system

Abstract

The invention relates to the technical field of financial knowledge management, and discloses a multi-modal Agent coordinated reasoning system, and the key points of the technical scheme are that the multi-modal Agent coordinated reasoning system comprises a data interface module which is used for connecting a data source to obtain target data, and is also used for receiving a task instruction; the function management module is used for constructing functional Agents with various data processing functions and executing corresponding data processing on target data according to a scheduling strategy through the functional Agents; the business management module is used for constructing a business Agent of each business field and carrying out business processing according to a scheduling strategy through the business Agent; and the scheduling decision module is used for generating a scheduling strategy according to the task instruction and the target data, and performing scheduling and collaboration on the function Agent and the service Agent according to the scheduling strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of financial knowledge management, and more particularly, to a multimodal Agent coordinated reasoning system. Background Art

[0002] In the traditional knowledge management field, knowledge management currently mostly utilizes databases and knowledge bases, document management systems, content management systems, CRM, big data platforms, business systems, etc. Although traditional technologies are still the "infrastructure" of knowledge management in most enterprises, in the context of consumer finance, the traditional knowledge management system has the following problems in daily business management:

[0003] Limitations of single-modal data processing: Current knowledge management systems in the consumer finance field mostly rely on a single text data processing method, which makes it difficult to effectively integrate multimodal information such as images, voice, and video, resulting in one-sided knowledge expression and insufficient information utilization.

[0004] Limited reasoning capabilities: The reasoning mechanism in traditional knowledge management systems is often based on a single agent and lacks cross-modal and cross-domain collaborative reasoning capabilities, making it difficult to cope with complex and changing market environments and customer needs.

[0005] Delayed knowledge updating: Knowledge in the consumer finance sector is updated extremely quickly, but the existing system lacks a real-time updating mechanism, resulting in outdated information in the knowledge base and an inability to provide timely and accurate support for decision-making.

[0006] Poor user experience: The existing system lacks user participation and feedback mechanisms in knowledge management and service provision, and cannot meet users' demand for personalized, high-quality financial services.

[0007] For example: Single-modal knowledge management system: Taking the text-based knowledge graph construction method as an example, although it realizes the structured representation of knowledge to a certain extent, it cannot integrate multimodal features, which limits the application scope and value of knowledge.

[0008] Isolated Agent Reasoning System: The lack of cross-agent collaboration mechanism leads to information isolation between agents, making it impossible to form effective knowledge sharing and collaborative reasoning, affecting the overall performance and efficiency of the system.

[0009] Static knowledge base system: It relies on manual updates, is inefficient and difficult to adapt to the rapidly changing market environment, and cannot meet the actual needs of the consumer finance field. Summary of the Invention

[0010] The purpose of this invention is to provide a multimodal agent coordinated reasoning system with more accurate decision-making, improved collaborative efficiency, strong dynamic adaptability, strong interpretability, and improved user experience.

[0011] The above technical objectives of the present invention are achieved through the following technical solutions: A multimodal agent coordinated reasoning system for financial knowledge management, comprising:

[0012] The data interface module is used to connect to the data source to obtain target data and also to receive task instructions;

[0013] Function management module is used to build functional agents with various data processing functions, and through each functional agent, according to the scheduling strategy, perform corresponding data processing on the target data;

[0014] The business management module is used to build business agents in various business areas and process business according to scheduling strategies through business agents;

[0015] The scheduling decision module is used to generate a scheduling strategy based on task instructions and target data, and schedule and coordinate functional agents and business agents according to the scheduling strategy.

[0016] As a preferred technical solution of the present invention, the functional agents include: data standardization agent, feature extraction agent, feature fusion agent;

[0017] The data standardization processing agent is used to clean, convert and standardize the target data to obtain standard data;

[0018] The feature extraction agent extracts features from the target data through an integrated deep learning model and aligns the extracted feature vectors in a unified semantic space across modalities.

[0019] The feature fusion agent fuses the features of the target data through a cross-modal attention mechanism.

[0020] As a preferred technical solution of the present invention, the business agent is used to construct a knowledge graph containing entities and their relationships based on the extracted multimodal feature information; to update and modify the content of the knowledge graph of the current business field; and to perform logical reasoning based on the knowledge graph.

[0021] As a preferred technical solution of the present invention, the scheduling decision module includes a multi-intelligent scheduling decision submodule, a distributed collaboration submodule, and a collaboration strategy optimization submodule;

[0022] The multi-intelligent scheduling submodule, based on the scheduling task knowledge base, according to task instructions and target data, uses a large model to optimize decision-making, decomposes tasks, obtains subtasks, and matches the subtasks with business agents and functional agents, and generates a dynamic scheduling strategy based on the matching results;

[0023] The distributed collaboration submodule is used to share local decisions and collaborate among multiple business agents and function agents according to the scheduling strategy;

[0024] The collaboration strategy optimization submodule is used to formulate the interaction mode and task allocation strategy between the business agent and the functional agent.

[0025] As a preferred technical solution of the present invention, it also includes an evaluation feedback module for constructing a feedback handling agent, which is used to obtain feedback responses, process and analyze the feedback responses, and update parameters of the function management module, business management module and scheduling decision module based on the analysis results.

[0026] The evaluation feedback module is provided with a consistency verification submodule, a repair submodule, and a closed-loop optimization submodule. The consistency verification submodule is used to mutually verify the consistency of the inference results through multimodal data; the repair submodule is used to repair data samples for the inference results that fail the consistency verification; the closed-loop optimization submodule is used to deploy a feedback optimization mechanism. Through the feedback optimization mechanism, the feedback response data is used as a learning signal and input into the system's training set to update the parameters in the system.

[0027] As a preferred technical solution of the present invention, it also includes a security protection module, which is provided with an identity authentication submodule, a transmission encryption submodule, a log recording submodule, and a virus protection submodule;

[0028] The identity authentication submodule is used to verify the identity and authority of the login personnel;

[0029] The transmission encryption submodule is used to encrypt and verify the legitimacy of the transmitted business data information;

[0030] The log recording submodule is used to record system operation logs;

[0031] The virus protection submodule is used to detect external data and intercept virus data.

[0032] As a preferred technical solution of the present invention, the functional agent includes: a text analysis agent, a visual reasoning agent;

[0033] The business agents include: credit approval decision agent, marketing decision agent, decision generation agent, and process processing agent.

[0034] In summary, the present invention has the following beneficial effects: The present invention proposes a multimodal agent reasoning framework for knowledge management in the consumer finance field, which achieves:

[0035] Precision decision-making: Design "horizontal and vertical" agent classification and integration. By designing "horizontal" functional agents, we can achieve knowledge classification management, deep mining and comprehensive integration by functional domain, and improve knowledge expression and reasoning capabilities in complex scenarios. By designing "vertical" business agents, we can achieve the management and use of thematic knowledge graphs by business field, and improve the decision-making accuracy of intelligent agents.

[0036] Improved collaborative efficiency: Design multiple intelligent scheduling and collaborative agents to self-organize and self-collaborate various "vertical" business agents and "horizontal" functional agents. Targeting different consumer finance tasks and multimodal inputs, this system enables automated multi-agent combination division of labor, scheduling decision-making, and feedback optimization, improving the efficiency and accuracy of knowledge utilization, reducing computing load, and increasing response speed.

[0037] Strong dynamic adaptability: Based on dynamic knowledge graph and real-time update mechanism, it ensures the timeliness and accuracy of knowledge and effectively responds to the rapidly changing market environment.

[0038] Strong explainability: Generating visual reasoning paths enhances users' trust in reasoning results and improves the transparency and credibility of the system.

[0039] Improve user experience: Improve user experience through result optimization and feedback mechanisms to meet users' needs for personalized, high-quality financial services. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a system architecture diagram of the present invention;

[0041] Figure 2 It is a flow chart of the technical solution of the present invention;

[0042] Figure 3 It is a device module diagram of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below with reference to the accompanying drawings.

[0044] The present invention provides a multimodal agent coordinated reasoning system for financial knowledge management. The system of the present invention is a "vertical" and "horizontal" knowledge management and domain intelligence agent segmentation framework, and is a multimodal agent reasoning framework solution for knowledge management in the consumer finance field.

[0045] This system includes: data interface module, function management module, business management module, scheduling decision module, and evaluation feedback module.

[0046] The data interface module is used to connect to the data source to obtain target data and also to receive task instructions to ensure high-quality data input for subsequent modules;

[0047] Function management module is used to build functional agents with various data processing functions, and through each functional agent, according to the scheduling strategy, perform corresponding data processing on the target data;

[0048] That is, to build a "horizontal" functional domain knowledge management and reasoning agent. Various functional agents integrate advanced data processing models and reasoning models of various subdivided functional domains to achieve professional analysis and processing integration of different types of data types.

[0049] Functional agents include: data standardization agent, feature extraction agent, and feature fusion agent. Functional agents can also set up special text analysis agents and visual reasoning agents according to actual use. There can be functional overlap between multiple functional agents.

[0050] Specifically, the data standardization agent is used to clean, convert, and standardize the target data to obtain standard data; the types of target data include text, images, voice, video, etc.; cleaning includes removing noise and outliers; format conversion is to convert images of different formats into a unified format.

[0051] The feature extraction agent extracts features from the target data through integrated deep learning models, such as the text analysis agent integrating the Transformer model and the visual processing agent integrating the ViT model. It also uses contrastive learning frameworks such as CLIP to align the extracted feature vectors in a unified semantic space across modalities using a contrastive loss function.

[0052] The feature fusion agent uses a cross-modal attention mechanism to fuse the features of the target data. Specifically, the model focuses on relevant content in data from different modalities, achieving effective fusion of features. For example, in the consumer finance field, it can perform correlation analysis between user-submitted text descriptions and uploaded income certificate images.

[0053] The business management module is used to build business agents in various business areas and process business according to scheduling strategies through business agents;

[0054] The business agent is used to construct a knowledge graph containing entities and their relationships based on the extracted multimodal feature information. Entities include customers, products, services, etc., thereby building a thematic knowledge base for the current business field. When constructing the knowledge graph, a timestamp is used to record the update time of each node, and a confidence weight is set to reflect the reliability of the information source.

[0055] When the business agent builds the knowledge graph, it also provides storage and query functions for the knowledge graph, supports high-level semantic search and intelligent recommendation, and can help users quickly obtain the required information.

[0056] The business agent is used to update and modify the knowledge graph for the current business domain. Specifically, an incremental update mechanism is designed to support the rapid integration of new information and the timely modification of old information, enabling real-time updates of the knowledge graph for the business domain. For example, when a new credit policy is released, the system can automatically adjust the information of the relevant nodes.

[0057] Business agents are used to perform logical reasoning based on knowledge graphs. Specifically, they use technologies such as graph neural networks to mine deep relationships from knowledge graphs and support complex logical reasoning. For example, they can identify potential risk factors or recommend personalized financial products.

[0058] Business agents include: credit approval decision agent, marketing decision agent, decision generation agent, and process processing agent.

[0059] The scheduling decision module generates scheduling strategies based on task instructions and target data, and schedules and coordinates functional and business agents based on these strategies. Specifically, the scheduling decision module is used to build multiple intelligent scheduling and collaboration agents, enabling the self-organization and self-coordination of various "vertical" business agents and "horizontal" functional agents. It features flexible task scheduling and conflict resolution mechanisms, ensuring robust performance even in highly concurrent environments.

[0060] The scheduling decision module includes a multi-intelligent scheduling decision submodule, a distributed collaboration submodule, and a collaborative strategy optimization submodule;

[0061] The multi-intelligent scheduling submodule uses a large model to optimize decision-making based on the scheduling task knowledge base, task instructions, target data, including attachments, etc., to decompose tasks into subtasks. These subtasks are then matched with business agents and functional agents. Based on the matching results, a dynamic scheduling strategy combining machine learning algorithms and heuristic algorithms is used to generate dynamic scheduling strategies based on the real-time task complexity and resource load to implement scheduling decisions.

[0062] The distributed collaboration submodule is used to share local decisions and collaborate across multiple business and functional agents based on scheduling strategies. Specifically, it uses federated learning or consensus algorithms to ensure that multiple agents can efficiently share local decisions and work together. For example, when evaluating a loan application, a text analysis agent processes personal information, a visual reasoning agent analyzes financial documents, and a credit approval decision agent combines the results of both to make the final decision.

[0063] The collaborative strategy optimization submodule introduces a reinforcement learning algorithm to formulate the interaction mode and task allocation strategy between business agents and functional agents. As the environment changes or business needs are adjusted, the system can automatically adapt and improve overall efficiency.

[0064] The evaluation feedback module is used to build the feedback handling agent, which is used to obtain feedback responses, process and analyze the feedback responses, and update the parameters of the function management module, business management module and scheduling decision module based on the analysis results.

[0065] Specifically, the evaluation and feedback module builds an evaluation, feedback and optimization mechanism for the intelligent agent's decision results, supports autonomous feedback responses based on preset quantitative evaluation indicators, and manual feedback responses based on user page input, realizes the processing and analysis of feedback information, and optimizes the knowledge system and model parameters.

[0066] The evaluation feedback module is provided with a consistency check submodule, a repair submodule, and a closed-loop optimization submodule;

[0067] The consistency check submodule is used to verify the consistency of inference results through multimodal data to reduce misjudgments. For example, it checks whether a user's credit score is consistent with their historical transaction records.

[0068] The repair submodule is used to repair data samples for inference results that fail consistency checks. Specifically, GAN repair is used. For inaccurate results, adversarial generative networks (GANs) are used to generate data samples that are more consistent with the actual situation, thereby correcting model deviations and improving prediction accuracy.

[0069] The closed-loop optimization submodule is used to deploy a feedback optimization mechanism. Through the feedback optimization mechanism, the feedback response data is used as a learning signal and input into the system's training set to update the parameters in the system, continuously improving the knowledge graph and the agent's behavior rules to better meet user needs.

[0070] During actual deployment, this system also deploys a security protection module and a visualization page on the device. The visualization page is used to provide an intuitive interface to display the inference results and support user interaction with the system, such as adjusting parameter settings by dragging components or directly marking information points that require special attention on the interface.

[0071] The security protection module is provided with an identity authentication submodule, a transmission encryption submodule, a log recording submodule, and a virus protection submodule;

[0072] The identity authentication submodule is used to verify the identity and authority of the login personnel, and can strictly limit the access rights of relevant personnel to the system hardware and software. Entry to the system must undergo strict password checking and identity authentication;

[0073] The transmission encryption submodule is used to encrypt and verify the legitimacy of transmitted business data information. Specifically, business data information transmission must be encrypted and secured, and legally and technically verified;

[0074] The logging submodule is used to record system operation logs;

[0075] The virus protection submodule is used to detect external data, intercept virus data, and prevent virus intrusion.

[0076] In a specific embodiment of the present invention, the business agents include decision-making agents and process processing agents, the functional agents include text analysis agents and visual reasoning agents, the evaluation and feedback module utilizes a feedback processing agent, and the scheduling decision module is implemented as a multi-intelligent scheduling decision maker. Based on consumer finance domain knowledge, the multi-intelligent scheduling and collaboration agents implement autonomous multi-agent organization and collaboration, decision-making, and processing based on diverse consumer finance domain tasks and multimodal input.

[0077] System architecture such as Figure 1 As shown in the figure, a modular layered design is adopted, and each agent interacts through a standardized interface and a unified data format, supporting dynamic expansion and load balancing.

[0078] The system's operating process is as follows Figure 2 As shown; specifically includes the following steps:

[0079] 1. User input is performed through the data interface module.

[0080] User input includes user command prompts and mixed multimodal attachments, which can be unstructured documents, images, videos, audio, etc. For example, a customer submits the following credit application materials through a bank's online platform:

[0081] Credit application form: contains text information such as personal information, income, credit history, etc.

[0082] Financial documents: scanned copies or images of payslips, bank statements, tax bills, etc.;

[0083] Video proof: such as customer-recorded environmental videos (workplace, asset status), identity verification videos (such as "blinking" or "reading a number" action verification).

[0084] 2. Based on the user input content, the scheduling decision module makes multi-intelligent scheduling decisions.

[0085] As the core coordination module of the system, the multi-intelligent scheduling decision maker adopts an intelligent task allocation algorithm based on the combination of machine learning and heuristic algorithms to schedule agents in real time according to the input type (text, image, mixed) and task complexity.

[0086] Its implementation includes:

[0087] 2.1. Input type recognition: Quickly determine the input modality through pre-trained multimodal classification models (such as CLIP);

[0088] 2.2 Task priority assessment: Dynamically assign weights based on the urgency of user instructions (such as keyword matching) and system resource status (such as GPU utilization);

[0089] 2.3. Agent status monitoring: Use a heartbeat detection mechanism (such as a Redis-based distributed lock) to track the availability of each agent in real time to ensure fault tolerance and load balancing.

[0090] 3. Perform text analysis through the corresponding functional agent.

[0091] The text analysis agent uses a deep learning-driven feature extraction and semantic understanding framework, which includes:

[0092] 3.1 Text feature extraction:

[0093] Preprocessing: Use regular expressions and word segmentation tools (such as Jieba) to remove noise, filter stop words, and extract task-related data;

[0094] Feature encoding and multimodal context fusion: Generate text vectors through a pre-trained Transformer model, and perform weighted fusion of the text vectors and the intent vectors of the task instructions through an attention mechanism (such as Scaled Dot-Product Attention) to generate a comprehensive semantic representation.

[0095] For example, structured data such as income and debt ratio, as well as text descriptions such as loan purpose and customer address, can be extracted from a credit application form and analyzed based on a large prediction model to obtain comprehensive semantic analysis results.

[0096] 3.2 Knowledge base retrieval and tool calling:

[0097] Subject knowledge base retrieval: Based on the constructed prompt, relevant documents (such as banking policies and legal and regulatory documents) are retrieved from the subject knowledge base (such as the structured knowledge graph built by Elasticsearch) through a vector database (such as Chromadb);

[0098] External tool call: Dynamically call third-party tools through the API gateway to support real-time data query (such as credit business database, log system, related transaction system, etc.).

[0099] 3.3 Preliminary Approval and Decision Triggering:

[0100] Initial Approval: Conduct preliminary approval analysis by searching internal and external information and knowledge (e.g., searching for relevant terms and conditions, screening applicants based on credit score thresholds and anti-money laundering regulations).

[0101] Decision triggering: Based on the output confidence score, the agent autonomously decides to trigger interactions with other agents and execute subsequent processes. (For example, based on clues from text retrieval and input task type, the text analysis results are pushed to the cross-modal feature fusion engine.)

[0102] 4. Perform visual reasoning through the function agent of the function management module.

[0103] The visual reasoning agent uses multimodal fusion and fine-grained image understanding technology to achieve intelligent analysis of image input. Its implementation includes:

[0104] 4.1. Image data classification preprocessing:

[0105] Image processing: Use OpenCV to denoise, enhance, and crop image files to fit the input size (e.g., 224×224).

[0106] Perform OCR on uploaded financial documents (such as bank statement screenshots) to extract key values ​​(such as monthly income and account balance).

[0107] Video processing: Use FFmpeg to decode the video and extract key frames (such as facial movements during customer authentication and panoramic views of the work environment).

[0108] 4.2 Dynamic feature analysis:

[0109] Action recognition: Identify actions in videos (such as the customer's proficiency in operating the device and abnormal behavior in the environment) through models such as TCN (Temporal Convolutional Networks).

[0110] Feature encoding: Extracting visual feature vectors from input materials through the ViT (Vision Transformer) architecture model;

[0111] Material Verification: Using models such as the ResNet model, we can detect whether the file has been tampered with (e.g., watermarks, splicing traces). We can also perform action recognition (e.g., "blinking" and "nodding") on the video to verify identity authenticity.

[0112] Anomaly marking: Anomalies in the extracted structured information (such as large transfers and frequent withdrawals) are marked based on a large language model. Anomalies in structured information (such as fraudulent behaviors such as unnatural movements, repeated or synthesized environments) are marked based on time series models such as LSTM.

[0113] 5. Perform cross-modal feature fusion through the functional agent of the functional management module.

[0114] 5.1. Multimodal alignment: Map text vectors and visual vectors into a unified space through a cross-modal attention mechanism (such as CLIP's contrastive learning framework) (e.g., semantically matching the "vehicle model" detected in the video with the "asset description" in the application form);

[0115] 5.2. Fine-grained reasoning: Use graph neural networks (GNNs) to build visual-text relationship graphs and infer implicit semantics (for example, in remote electronic verification services, establish relationships between users' facial features and language text features to infer hidden fraud risks).

[0116] 6. Make decisions through the business management module.

[0117] The decision-making agent is based on a hybrid reasoning model of reinforcement learning and knowledge-driven to achieve dynamic decision-making:

[0118] 6.1 Knowledge base retrieval and model optimization:

[0119] Decision knowledge base retrieval: Query historical decision cases through the constructed decision tree model (such as XGBoost) or knowledge graph (such as Neo4j) (for example, using the fused feature vector as input to retrieve similar customer use cases);

[0120] Model selection strategy: Dynamically select the optimal model based on task type, data distribution, and task timeliness requirements (such as LSTM for time series decision-making and Transformer for complex relational reasoning).

[0121] 6.2 Decision Reasoning and Evaluation

[0122] Multi-objective optimization: Based on matched similar historical cases, the system uses a step-by-step reasoning incentive mechanism based on a reinforcement learning model, as well as multi-objective trade-off mechanisms such as the NSGA-II algorithm, to optimize multiple decision-making objectives (such as cost, efficiency, and risk).

[0123] Quality assessment mechanism: Ensure decision reliability based on confidence thresholds (such as >0.8) and anomaly detection, and dynamically adjust strategies and hyperparameters based on subsequent feedback.

[0124] 6.3. Decision result output: The decision conclusion and processing task type information are output in a structured format (such as XML) and support interpretability through metadata annotation (such as the feature weights based on which the decision was made).

[0125] 7. Process the process through the business management module.

[0126] The process processing agent uses an event-driven workflow engine to achieve automated task execution:

[0127] 7.1. Workflow Generation:

[0128] Decision decomposition: Analyze and decompose the decision conclusions and determine the types of post-decision processing tasks (such as "approval report generation", "data analysis and statistics", "marketing strategy formulation", etc.);

[0129] Process engine: Based on historical experience and large-scale model inference, it dynamically generates task processing workflows, and executes parallel / serial and cyclic processes (for example, the "approval report generation" task is decomposed into: "data extraction → template selection → prompt word creation → plug-in call → large-scale model service call → report generation → visualization → email creation → report sending");

[0130] 7.2. Implementation Monitoring and Evaluation:

[0131] Real-time log analysis: Based on RAG processes and business operation knowledge, it enables intelligent collection, analysis, and monitoring of logs during process execution;

[0132] Quality Assessment: Execution snapshots of each workflow node are collected online and then evaluated offline. Functional metrics are determined by the node task type (e.g., accuracy and recall for classification decision tasks). Performance metrics include execution efficiency, throughput, and cost. The assessment conclusions provide knowledge support for subsequent workflow generation and optimization.

[0133] 8. Conduct feedback disposal through the evaluation feedback module.

[0134] The Feedback Processing Agent uses adaptive learning and knowledge graph update mechanisms to achieve continuous system optimization, including:

[0135] 8.1、User Feedback Processing:

[0136] Visualization generation: using D3.js to transform decision results into interactive charts;

[0137] Feedback classification: Determine the positivity or negativity of user feedback through sentiment analysis models.

[0138] 8.2. Knowledge Update and Review Optimization:

[0139] Positive feedback: Store the feature vectors and decision paths of successful cases into a knowledge graph (such as Neo4j) and update node relationships through graph embedding;

[0140] Negative feedback: Analyze failure cases through meta-learning, adjust knowledge graphs and retrieval strategies, model hyperparameters, and decision rules (such as regularly using A / B testing to compare the approval accuracy of new and old models and optimize video processing algorithms).

[0141] As a specific embodiment of the present invention, Figure 3 As shown, when the device is actually deployed in the present invention, it is equipped with a user interface module, a function management module, a business management module, a scheduling decision module, and an evaluation feedback module. The function management module is configured as a multimodal large model service module; the business management module is configured as: a domain knowledge base management module, a domain tool and plug-in agent module, a domain professional model service module, a decision and evaluation module, and a process processing module; and the evaluation decision module is configured as a feedback processing module. The application functions of each module are as follows:

[0142] User Interface Module: This module provides a visual human-machine interface for users and a RESTful service call interface for upper-layer applications. This module also allows users to perform operations related to result feedback.

[0143] The Scheduling Decision Module is responsible for real-time agent scheduling based on input type (text, image, mixed) and task complexity using a dynamic task allocation algorithm. It includes functions such as input type identification, task priority assessment, and agent status monitoring. A pre-trained multimodal classification model is used to quickly determine the input modality and dynamically assign weights based on the urgency of user commands and system resource status. Furthermore, a heartbeat detection mechanism is used to track the availability of each agent in real time to ensure system fault tolerance and load balancing.

[0144] Domain Knowledge Base Management Module: This module is primarily responsible for building, managing, and retrieving knowledge bases in the consumer finance sector. It includes a subject knowledge base retrieval function, which can retrieve relevant documents from a structured knowledge graph based on a vector database. It also supports external tool invocation, dynamically accessing third-party tools through an API gateway for real-time data queries.

[0145] Domain Tools and Plugin Proxy Module: This module provides the system with access to various professional tools and plugins, enabling the system to flexibly expand its functionality. For example, it can integrate credit business databases and logging systems through the API gateway to support a wider range of data processing and analysis needs.

[0146] Multimodal Large Model Service Module: This module integrates multimodal feature extraction and fusion technologies, supporting in-depth analysis of various data types, including text, images, and videos. It encapsulates large model services, extracting useful information from multiple data sources and fusing this information through a cross-modal attention mechanism to generate comprehensive semantic representations.

[0147] Domain-Specific Model Service Module: This module focuses on the specific needs of different business areas and provides customized model services. For example, it uses time series data analysis models, classification decision models, and knowledge graphs to meet different task requirements such as risk assessment and credit scoring.

[0148] The Decision and Evaluation Module implements a dynamic decision-making process based on a hybrid reasoning model of reinforcement learning and knowledge-driven reasoning. It includes functions such as decision knowledge base retrieval, model selection strategies, decision reasoning, and evaluation. This module also employs multi-objective optimization methods and quality assessment mechanisms to ensure the reliability and effectiveness of decisions.

[0149] The process processing module utilizes an event-driven workflow engine to automate task execution. It parses decision conclusions into specific processing tasks and dynamically generates task processing workflows based on historical experience. It also includes real-time log analysis and quality assessment capabilities to monitor and optimize process execution.

[0150] Feedback Processing Module: This module implements adaptive learning and knowledge graph update mechanisms to promote continuous system optimization. This module encompasses two aspects: user feedback processing and knowledge updating and replay optimization. Decision results are displayed through visualization tools, and system behavior is adjusted based on user feedback. For positive feedback, node relationships in the knowledge graph are updated; for negative feedback, the cause of failure is analyzed and model parameters and decision rules are adjusted accordingly.

[0151] The technical solution of the present invention integrates advanced deep learning technologies, cross-modal data processing methods, and dynamic knowledge graph construction strategies to effectively integrate and analyze diverse data sources in complex information environments. By detailedly segmenting consumer finance business and functional domains, a "vertical and horizontal" knowledge graph and domain agent segmentation scheme is proposed. This system automatically adjusts its behavior, evaluates decisions, and autonomously iterates and optimizes the agent's output based on the input data type and task requirements. Efficient communication protocols and conflict resolution mechanisms ensure the consistency and efficiency of task execution. Furthermore, the entire system design emphasizes the real-time collection and application of user feedback, forming a self-optimizing, continuously evolving closed-loop system. This comprehensive solution not only improves the automation level and accuracy of financial service processes such as credit approval, but also enhances the system's adaptability and robustness, enabling it to remain competitive in a rapidly changing market environment. Through this multi-layered, modular architectural design, the present invention demonstrates how to leverage modern artificial intelligence technologies to build a flexible and powerful multi-agent reasoning platform to address the growing challenges of the financial business.

[0152] The technical advantages of the present invention are:

[0153] Precision decision-making: Design "horizontal and vertical" agent classification and integration. By designing "horizontal" functional agents, we can achieve knowledge classification management, deep mining and comprehensive integration by functional domain, and improve knowledge expression and reasoning capabilities in complex scenarios. By designing "vertical" business agents, we can achieve the management and use of thematic knowledge graphs by business field, and improve the decision-making accuracy of intelligent agents.

[0154] Improved collaborative efficiency: Design multiple intelligent scheduling and collaborative agents to self-organize and self-collaborate various "vertical" business agents and "horizontal" functional agents. Targeting different consumer finance tasks and multimodal inputs, this system enables automated multi-agent combination division of labor, scheduling decision-making, and feedback optimization, improving the efficiency and accuracy of knowledge utilization, reducing computing load, and increasing response speed.

[0155] Strong dynamic adaptability: Based on dynamic knowledge graph and real-time update mechanism, it ensures the timeliness and accuracy of knowledge and effectively responds to the rapidly changing market environment.

[0156] Strong explainability: Generating visual reasoning paths enhances users' trust in reasoning results and improves the transparency and credibility of the system.

[0157] Improve user experience: Improve user experience through result optimization and feedback mechanisms to meet users' needs for personalized, high-quality financial services.

[0158] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal agent-based coordinated reasoning system for financial knowledge management, characterized by: include: The data interface module is used to connect to the data source to obtain target data and also to receive task instructions; Function management module is used to build functional agents with various data processing functions, and through each functional agent, according to the scheduling strategy, perform corresponding data processing on the target data; The business management module is used to build business agents in various business areas and process business according to scheduling strategies through business agents; The scheduling decision module is used to generate a scheduling strategy based on task instructions and target data, and schedule and coordinate functional agents and business agents according to the scheduling strategy.

2. A multimodal agent coordinated reasoning system for financial knowledge management according to claim 1, characterized by: Agents include: data standardization agent, feature extraction agent, and feature fusion agent; The data standardization processing agent is used to clean, convert and standardize the target data to obtain standard data; The feature extraction agent extracts features from the target data through an integrated deep learning model and aligns the extracted feature vectors in a unified semantic space across modalities. The feature fusion agent fuses the features of the target data through a cross-modal attention mechanism.

3. The multimodal agent coordinated reasoning system for financial knowledge management according to claim 1, characterized in that: The business agent is used to construct a knowledge graph containing entities and their relationships based on the extracted multimodal feature information; and is used to update and modify the content of the knowledge graph in the current business field; Used for logical reasoning based on knowledge graphs.

4. The multimodal agent coordinated reasoning system for financial knowledge management according to claim 1, characterized in that: The scheduling decision module includes a multi-intelligent scheduling decision submodule, a distributed collaboration submodule, and a collaboration strategy optimization submodule; The multi-intelligent scheduling submodule, based on the scheduling task knowledge base, according to task instructions and target data, uses a large model to optimize decision-making, decomposes tasks, obtains subtasks, and matches the subtasks with business agents and functional agents, and generates a dynamic scheduling strategy based on the matching results; The distributed collaboration submodule is used to share local decisions and collaborate among multiple business agents and function agents according to the scheduling strategy; The collaboration strategy optimization submodule is used to formulate the interaction mode and task allocation strategy between the business agent and the functional agent.

5. The multimodal agent coordinated reasoning system for financial knowledge management according to claim 1, characterized in that: It also includes an evaluation feedback module for building a feedback handling agent, which is used to obtain feedback responses, process and analyze the feedback responses, and update parameters of the function management module, business management module and scheduling decision module based on the analysis results. The evaluation feedback module is provided with a consistency check submodule, a repair submodule, and a closed-loop optimization submodule. The consistency check submodule is used to mutually verify the consistency of the reasoning results through multimodal data; The repair submodule is used to repair data samples for inference results that fail consistency verification; the closed-loop optimization submodule is used to deploy a feedback optimization mechanism. Through the feedback optimization mechanism, the feedback response data is used as a learning signal and input into the system's training set to update the parameters in the system.

6. The multimodal agent coordinated reasoning system for financial knowledge management according to claim 1, characterized in that: It also includes a security protection module, which is provided with an identity authentication submodule, a transmission encryption submodule, a log recording submodule, and a virus protection submodule; The identity authentication submodule is used to verify the identity and authority of the login personnel; The transmission encryption submodule is used to encrypt and verify the legitimacy of the transmitted business data information; The log recording submodule is used to record system operation logs; The virus protection submodule is used to detect external data and intercept virus data.

7. The multimodal agent coordinated reasoning system for financial knowledge management according to claim 1, characterized in that: The functional agents include: text analysis agent, visual reasoning agent; The business agents include: credit approval decision agent, marketing decision agent, decision generation agent, and process processing agent.

Images