A multi-modal intelligent interaction method based on a large language model

Abstract

The application discloses a multimodal intelligent interaction method based on a large language model, and the interaction method comprises the following steps: receiving multimodal input of a user; analyzing the user's intention, decomposing a complex task into multiple subtasks, and coordinating the execution of various professional tools; processing multiple types of input data; integrating various financial professional analysis tools and services; managing and maintaining the knowledge base of the system, dynamically updating and real-time searching knowledge; and providing the infrastructure required for system operation. Through automatic task decomposition and tool calling, the manual maintenance workload is reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal intelligent interaction method based on a large language model. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have been widely applied in various fields. In the financial investment field, intelligent investment advisory systems are gradually becoming an important tool for improving service efficiency and quality. Traditional intelligent investment advisory systems are mainly based on rule engines and simple machine learning models, providing investment consulting services to users through preset dialogue processes and fixed knowledge bases.

[0003] In existing technologies, intelligent interactive systems mainly adopt a single-modal interaction method, such as plain text dialogue or simple graphic display.

[0004] These systems have the following limitations when dealing with complex financial investment problems: First, they cannot effectively handle multimodal inputs, such as screenshots of user-uploaded holdings and complex financial charts; second, they lack the ability to intelligently decompose and coordinate complex tasks, making it difficult to handle comprehensive investment analysis involving multiple steps; and third, the systems have high costs for knowledge updates and maintenance, making it difficult to adapt to the rapidly changing financial market environment.

[0005] Most existing technologies remain limited to simple question-and-answer models, lacking in-depth financial expertise and multimodal processing capabilities. Especially when dealing with scenarios involving multiple data types and complex analytical tasks, existing technologies struggle to provide accurate and comprehensive investment advice.

[0006] In the prior art, the technical solution closest to the present invention includes: These existing technological solutions generally suffer from the following problems when dealing with complex financial investment scenarios: lack of multimodal data fusion and processing capabilities, inability to intelligently decompose complex tasks, lack of integration of professional financial analysis tools, and insufficient system scalability and adaptability.

[0007] The shortcomings of existing technologies include: 1. Limited Interaction Modality: The existing system mainly supports text interaction and cannot effectively handle multimodal data such as images, charts, and screenshots uploaded by users, which limits the application scenarios and user experience of the system.

[0008] 2. Limited task processing capacity: Existing technologies lack intelligent task decomposition and planning capabilities, making it difficult to handle complex investment analysis tasks involving multiple steps and tools.

[0009] 3. Low integration of professional knowledge: Existing systems typically rely on general language models and lack the integration of in-depth financial expertise and professional analysis tools, resulting in insufficient professionalism and accuracy in investment advice.

[0010] 4. Poor system scalability: Existing technical solutions typically adopt a monolithic architecture, making it difficult to flexibly integrate new analysis tools and knowledge sources, resulting in poor system scalability and maintainability.

[0011] 5. Weak contextual understanding: Existing systems have limited contextual understanding and memory capabilities in multi-turn dialogues, making it difficult to maintain long-term, coherent dialogue and analysis processes. Summary of the Invention

[0012] In view of the above problems, the present invention is proposed to provide a multimodal intelligent interaction method based on a large language model to overcome or at least partially solve the above problems.

[0013] According to one aspect of the present invention, a multimodal intelligent interaction method based on a large language model is provided, the interaction method comprising: Receive multimodal input from users; Analyze user intent, break down complex tasks into multiple sub-tasks, and coordinate the execution of various specialized tools; Processes various types of input data; Integrates a variety of professional financial analysis tools and services; Manage and maintain the system's knowledge base, and perform dynamic updates and real-time retrieval of knowledge; Provide the infrastructure required for the system to operate.

[0014] Optionally, receiving multimodal input from the user specifically includes text, images, and charts, and presenting the analysis results to the user.

[0015] Optionally, the analysis of user intent, the decomposition of complex tasks into multiple sub-tasks, and the coordination of the execution of various specialized tools specifically include: The core component is the task planning agent, which analyzes user intent, breaks down complex tasks into multiple sub-tasks, and coordinates the execution of various specialized tools.

[0016] Optionally, processing multiple types of input data specifically includes: Text understanding, image recognition, OCR processing, and chart parsing; It integrates multiple professional processing models, including entity extraction model, amount extraction model, and OCR recognition model.

[0017] Optionally, the integration of multiple financial professional analysis tools and services specifically includes: quantitative analysis models, risk assessment models, asset allocation engines, and market data analysis services.

[0018] Optionally, the knowledge base of the management and maintenance system specifically includes: a financial knowledge graph, historical dialogue records, and user profile data.

[0019] Optionally, the infrastructure required for system operation includes: database, cache, message queue, and monitoring.

[0020] This invention provides a multimodal intelligent interaction method based on a large language model. The interaction method includes: receiving multimodal input from users; analyzing user intent, decomposing complex tasks into multiple sub-tasks, and coordinating the execution of various professional tools; processing various types of input data; integrating multiple financial professional analysis tools and services; managing and maintaining the system's knowledge base, performing dynamic updates and real-time retrieval of knowledge; and providing the infrastructure required for system operation. Through automated task decomposition and tool invocation, the workload of manual maintenance is reduced.

[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 The flowchart illustrates a multimodal intelligent interaction method based on a large language model, as provided in an embodiment of the present invention. Detailed Implementation

[0024] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0025] The terms "comprising" and "having," and any variations thereof, in the specification, embodiments, claims, and drawings of this invention are intended to cover non-exclusive inclusion, such as including a series of steps or units.

[0026] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0027] like Figure 1 As shown, a multimodal intelligent interaction method based on a large language model is proposed. The interaction method includes: receiving multimodal input from users; analyzing user intent, decomposing complex tasks into multiple sub-tasks, and coordinating the execution of various professional tools; processing various types of input data; integrating various financial professional analysis tools and services; managing and maintaining the system's knowledge base, performing dynamic updates and real-time retrieval of knowledge; and providing the infrastructure required for system operation.

[0028] This invention provides a multimodal intelligent interaction system based on a large language model. The system adopts a layered architecture design and includes the following core components: 1. User interaction layer: This layer is responsible for receiving multimodal input from users, including text, images, charts, etc., and presenting the system's analysis results to the user in an appropriate format. This layer includes a front-end interface, API interfaces, and a response formatting module.

[0029] 2. Task Planning and Coordination Layer: The core component is the task planning agent, which analyzes user intent, breaks down complex tasks into multiple sub-tasks, and coordinates the execution of various specialized tools. The layers include an intent recognition module, a task decomposition module, a tool selection module, and an execution coordination module.

[0030] 3. Multimodal processing layer: This layer is responsible for processing different types of input data, including text understanding, image recognition, OCR processing, and chart parsing. It integrates various specialized processing models, such as entity extraction models, monetary value extraction models, and OCR recognition models.

[0031] 4. Professional Knowledge Service Layer: It integrates a variety of professional financial analysis tools and services, including quantitative analysis models, risk assessment models, asset allocation engines, and market data analysis services. Each tool provides services through standardized interfaces.

[0032] 5. Knowledge Management Layer: This layer is responsible for managing and maintaining the system's knowledge base, including financial knowledge graphs, historical dialogue records, and user profile data. It supports dynamic updates and real-time retrieval of knowledge.

[0033] 6. Basic support layer: Provide the infrastructure required for system operation, including databases, caches, message queues, and monitoring.

[0034] 1. Intelligent task decomposition mechanism of the task planning agent: By analyzing user input through a large language model, the system identifies task type and complexity, and automatically breaks down complex tasks into a sequence of executable subtasks. For example, when a user asks, "Please analyze the risk of my investment portfolio," the system breaks it down into subtasks such as: identifying holdings information → risk assessment → generating optimization suggestions.

[0035] 2. Multi-Agent Collaborative Working Mechanism: The system adopts a multi-agent architecture, including a task planning agent, an entity extraction agent, a data analysis agent, and an answer fusion agent. These agents communicate and collaborate through a standardized messaging protocol, enabling distributed processing of complex tasks.

[0036] 3. Large and small model collaboration mechanism: The system adopts a collaborative working mode of large and small models. The large model is responsible for task planning and complex reasoning, while the small model is responsible for specific tasks such as entity recognition and data extraction, thus achieving a balance between efficiency and effectiveness.

[0037] 4. Professional financial instrument integration mechanism: The system integrates a variety of professional financial analysis tools, including: IndicatorRecall: Handles specific quantitative standards or data queries involving stocks, funds, and bonds; MultIndicatorRecall: Analyzes fund performance using multi-dimensional key indicators; CampisiRecall: Specifically designed for attribution analysis of returns for pure bond funds; BrinsonRecall: Analyzing Excess Returns of Equity / Mixed Funds, Deconstructing Asset Allocation and Target Selection Effects; FundStyleRecall: Morningstar Style Box Fund Analysis Tool; SecurytiesMovement: A securities model used to detect abnormal fluctuations and assess corporate risk; StockBaseRecall: Analyze company financial statements and evaluate financial indicators; Dialogue-based stock selection agent: Generates SQL statements based on user queries to retrieve data. Example 1: Intelligent Portfolio Analysis Scenario When a user uploads a screenshot of their holdings and asks, "Please analyze my investment portfolio for me": 1. The system receives image input, and the OCR agent identifies the holding information; 2. Task planning agent decomposes tasks: position identification → risk assessment → configuration optimization; 3. Each agent works collaboratively, calling relevant analysis tools, such as StockBaseRecall to analyze the fundamentals of the stocks held, and SecuritiesMovement to detect abnormal fluctuations; 4. The answer integrates the agent's analysis results to generate a comprehensive report.

[0038] Example 2: Multi-turn dialogue scenario: When users conduct multiple rounds of investment consultations: 1. The context management module maintains the dialogue history; 2. The task planning agent understands user intent based on context; 3. Dynamically invoke relevant analysis tools, such as IndicatorRecall to obtain technical indicator data, and FundStyleRecall to analyze fund styles; 4. Maintain the consistency and coherence of the interaction.

[0039] Example 3: Real-time Market Analysis: A user asked: "How did the tech sector perform today? Are there any investment opportunities?" 1. The entity recognition module identifies entities in the "Technology" section; 2. The dialog-based stock selection agent generates SQL queries to retrieve stock data for the technology sector; 3. MultIndicatorRecall analyzes multi-dimensional indicators for stocks within a sector; 4. The reasoning enhancement module combines information from multiple sources to discover investment opportunities.

[0040] Beneficial effects: Improved processing efficiency: Through intelligent task decomposition and multi-agent collaboration, the system improves the efficiency of handling complex financial analysis tasks by more than 60% compared to traditional methods.

[0041] Enhanced interactive experience: Supports multimodal interaction, allowing users to interact with the system through text, images, and other means, improving interaction satisfaction by 45%.

[0042] Improved analytical accuracy: By integrating professional analytical tools and knowledge graphs, the accuracy of investment recommendations is improved by 80% compared to general dialogue systems.

[0043] Improved system scalability: Modular architecture design reduces the integration time of new features from 2-3 weeks in traditional systems to 2-3 days.

[0044] Reduce maintenance costs: By automating task breakdown and tool calls, the amount of manual maintenance work is reduced by about 70%.

[0045] Enhance professional competence: Deep integration with professional financial analysis tools improves the professionalism score of investment decisions by 85%.

[0046] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal intelligent interaction method based on a large language model, characterized in that, The interaction method includes: Receive multimodal input from users; Analyze user intent, break down complex tasks into multiple sub-tasks, and coordinate the execution of various specialized tools; Processes various types of input data; Integrates a variety of professional financial analysis tools and services; Manage and maintain the system's knowledge base, and perform dynamic updates and real-time retrieval of knowledge; Provide the infrastructure required for the system to operate.

2. The multimodal intelligent interaction method based on a large language model according to claim 1, characterized in that, The multimodal input received from the user specifically includes text, images, and charts, and the analysis results are presented to the user.

3. The multimodal intelligent interaction method based on a large language model according to claim 1, characterized in that, The process of analyzing user intent, breaking down complex tasks into multiple sub-tasks, and coordinating the execution of various specialized tools specifically includes: The core component is the task planning agent, which analyzes user intent, breaks down complex tasks into multiple sub-tasks, and coordinates the execution of various specialized tools.

4. The multimodal intelligent interaction method based on a large language model according to claim 1, characterized in that, The processing of various types of input data specifically includes: Text understanding, image recognition, OCR processing, and chart parsing; It integrates multiple professional processing models, including entity extraction model, amount extraction model, and OCR recognition model.

5. The multimodal intelligent interaction method based on a large language model according to claim 1, characterized in that, The integration of multiple financial professional analysis tools and services specifically includes: quantitative analysis models, risk assessment models, asset allocation engines, and market data analysis services.

6. The multimodal intelligent interaction method based on a large language model according to claim 1, characterized in that, The knowledge base of the management and maintenance system specifically includes: financial knowledge graph, historical dialogue records, and user profile data.

7. The multimodal intelligent interaction method based on a large language model according to claim 1, characterized in that, The infrastructure required for the system to operate specifically includes: database, cache, message queue, and monitoring.

Images