End-to-end agent-based financial modeling approach for multimodal financial analysis tasks

Abstract

The present invention discloses an end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks, which includes the following steps: constructing a multimodal financial dataset suitable for fine-tuning; fine-tuning a large language model using the constructed financial dataset; using a general large model as an intelligent agent model to identify the intent of user questions, and generating corresponding API request parameters or database retrieval parameters to rewrite the original knowledge documents obtained from the API request and database retrieval; reordering and reciprocal sorting of all acquired knowledge documents; combining the knowledge documents with financial task prompt words, inputting the fine-tuned financial large model, and performing effective financial analysis. The end-to-end intelligent agent financial large model method proposed in the present invention mainly solves the problems of low quality and weak relevance of knowledge documents retrieved from the database, and can effectively improve the signal-to-noise ratio of knowledge documents, thereby improving the factuality and reliability of the output content of the financial large model.

Description

Technical Field

[0001] The present invention relates to financial big model technology, and in particular to an end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks. Background Art

[0002] Financial analysis tasks primarily encompass two key areas: stock trend forecasting and financial question-answering. Machine learning and deep learning algorithms have been widely applied in time series forecasting, and large language models have also made significant progress in areas such as text sentiment analysis, text intent recognition, and open-domain dialogue. However, completing financial analysis tasks requires extensive domain-specific background knowledge and strong timeliness. Machine learning and deep learning algorithms cannot provide detailed reasoning for predictions, lacking interpretability. Large language models also exhibit limitations in practical applications, hindering their ability to keep up with the latest market trends and information. Retrieval-augmented generation techniques can effectively improve the real-time performance, reliability, and accuracy of large financial models.

[0003] In existing work, Yang et al. (FinGPT - Open-Source Financial Large Language Models) used a large language model-based approach to complete financial analysis tasks, leveraging the text generation capabilities of large language models to implement simple financial question-answering. However, this approach suffers from issues such as insufficient real-time performance and reliability. Xie et al. (PIXIU - A Large Language Model Instruction Data and Evaluation Benchmark for Finance) proposed a financial training dataset for training large language models to improve their financial understanding and analysis capabilities. However, this dataset primarily focuses on traditional natural language processing tasks such as sentiment analysis and text summarization, but is relatively weak in practical tasks such as stock price prediction and financial question-answering. To this end, Zhang et al. (Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models) used retrieval augmentation technology for financial analysis. By constructing a local financial real-time database, they retrieved the knowledge documents most relevant to the user's question from the local database when completing the financial analysis task. This knowledge served as external knowledge to assist the large financial model in providing responses, effectively addressing the large model's content hallucination and inability to generate real-time content. However, existing retrieval-enhanced generative techniques all use a single text-based document as external knowledge, such as news and research reports, and fail to mine other modal information with greater analytical value, such as market price time series data and market price indicator images. To address this, this paper proposes an end-to-end agent-based financial model approach. Summary of the Invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention proposes an end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks.

[0005] The present invention is achieved through at least one of the following technical solutions.

[0006] The end-to-end agent-based financial model approach for multimodal financial analysis tasks includes the following steps:

[0007] (a) Constructing a multimodal financial dataset;

[0008] (b) Fine-tuning financial models using multimodal financial datasets;

[0009] (c) Use the general large model as the agent model to identify the user's question intent and generate corresponding API request parameters or database retrieval parameters;

[0010] (d) Rewrite the original knowledge documents obtained from API requests and knowledge base retrieval;

[0011] (e) Re-rank and reverse-rank all acquired knowledge documents;

[0012] (f) Combine knowledge documents with financial task prompts and input them into the fine-tuned financial model to conduct effective financial analysis.

[0013] Furthermore, the multimodal financial dataset includes: a general financial dataset, a financial image analysis dataset, a financial news dataset, and a financial question-answering dataset;

[0014] Among them, the general financial dataset consists of community open source English financial datasets, which are used to enhance the information extraction and text summarization capabilities of large models;

[0015] Financial news and financial question-answering datasets are derived from publicly available stock price series and news on the Internet. They are used to expand the diversity of dataset samples and enhance the financial analysis capabilities of large models.

[0016] The financial image analysis dataset comes from publicly available stock price series data on the Internet and is used to enhance the large model's ability to understand the financial indicator information in the image.

[0017] Furthermore, the low-rank adaptive LoRA method is used to complete fine-tuning. During the fine-tuning process, the original parameters in the financial model will be frozen, and only the parameters in the newly added LoRA module will be updated.

[0018] Furthermore, the intelligent agent model uses the ChatGLM2-6B, Qwen-7B open source models, or uses API call models such as the ChatGPT model. By setting prompt words, the intelligent agent model determines whether it is necessary to request relevant knowledge documents and what type of knowledge documents to request by answering the current user question, and generates a JSON object of API request or database search parameters according to the format constrained by the prompt words:

[0019] (R,P)=Agent(Q)

[0020]

[0021] Among them, R and P represent the intention recognition result set and request parameter set of the intelligent agent model Agent for the user question Q, r j The value is 0 or 1, representing the use of API request or database retrieval, respectively. j For r jThe corresponding request parameter, d j Represents knowledge documents related to the user's question Q obtained through API requests or database retrieval.

[0022] Furthermore, in step (d), the ChatGPT model is used to rewrite the original documents, extract the documents and key information, and obtain fine-grained entity-level documents:

[0023] d′ j =LLM rewrite (d j )

[0024] Where d′ j Represents the fine-grained document after rewriting by the ChatGPT model, d j Represents knowledge documents related to the user's question Q obtained through API requests or database retrieval.

[0025] Furthermore, we use the BM25 algorithm to perform sparse search in the local database, and use the inverse sort fusion method to merge the results of vector search and sparse search. At the same time, we use the Cross-Encoder method to re-rank all knowledge documents obtained from API requests and database searches:

[0026] D=Rerank(d′1,d′2,...,d′ j )

[0027] Where D represents the re-ranked knowledge document set, d′ j Represents a fine-grained document after rewriting by the ChatGPT model.

[0028] Furthermore, the knowledge documents and user questions obtained in step (e) are input into the fine-tuned financial model for final financial analysis:

[0029] Res=LLM tuned (D, Q)

[0030] Among them, D represents the re-ranked knowledge document set, and Q represents the user question.

[0031] Furthermore, the financial model refers to any pre-trained general large language model.

[0032] Furthermore, the pre-trained general large language model is the Tongyi Qianwen Visual Understanding Model (Qwen-VL) and the Jiang Ziya-Multimodal Model.

[0033] Furthermore, the prompt words are constructed in text form, including role designation, task description, user question, reply format constraint, knowledge document reference part

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) By constructing multimodal financial datasets, efficiently fine-tuning large models, and injecting financial domain knowledge into financial large models, their financial analysis capabilities are enhanced;

[0036] (2) The agent model is introduced to improve the accuracy and relevance of knowledge document retrieval by leveraging the capabilities of large models;

[0037] (3) The quality of knowledge documents is further improved by using the fusion method of re-ranking and inverse ranking, so that the financial big model pays more attention to documents with higher relevance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of an end-to-end intelligent agent financial big model method for multimodal financial analysis tasks proposed in an embodiment of the present invention;

[0039] Figure 2 1. It is a schematic diagram of the framework structure of an end-to-end intelligent financial big model method for multimodal financial analysis tasks in an embodiment;

[0040] Figure 3 Schematic diagram of a method for constructing a multimodal financial dataset according to an embodiment;

[0041] Figure 4 is a schematic diagram of an image of financial indicators in an embodiment;

[0042] Figure 5 This is a comparison chart of the content generation effects of the main model of the embodiment and the ChatGPT model. DETAILED DESCRIPTION

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] like Figure 1 、 Figure 2 As shown, an end-to-end intelligent agent-based financial big model method for multimodal financial analysis tasks in this embodiment improves the signal-to-noise ratio of knowledge documents by introducing intelligent agent models and multimodal information, thereby enhancing the factuality and reliability of the output content of the big model. The method includes the following steps:

[0045] (a) Constructing a multimodal financial dataset suitable for fine-tuning. The training dataset is a key factor in determining the performance of large models. Currently, open-source training data suitable for general tasks is relatively abundant, but datasets targeting specific domains are relatively scarce. In particular, open-source datasets combining image and text data for the financial sector are scarce. To enhance the understanding and analysis capabilities of large models in the financial sector, this example uses open-source data to collect and organize a dataset suitable for fine-tuning large financial models and applying it to multimodal financial analysis tasks. These multimodal financial analysis tasks primarily involve multimodal financial data prediction and multimodal financial question-answering tasks. Multimodal financial analysis, as a key task, primarily covers two key areas: multimodal financial data prediction and multimodal financial question-answering. Currently, financial data prediction and financial question-answering methods primarily focus on market price time series data and financial text data. However, image data contains more financial indicator information and can better assist large financial models in completing financial analysis tasks. Therefore, multimodal financial data prediction and multimodal financial question-answering tasks are defined.

[0046] like Figure 3 As shown in Figure 1, the multimodal financial dataset consists of three parts: 1) general financial dataset; 2) financial news and financial question-answering dataset; 3) financial image analysis dataset.

[0047] The General Finance Dataset consists of community-sourced English financial datasets, including the Financial Phrase Library (FPB), Financial Question Answering (FinQA), Conversational Financial Question Answering (ConvFinQA), and Headline Financial News. This dataset is large in size, but relatively low in quality and has short labels. It can be used to enhance the information extraction and text summarization capabilities of large models.

[0048] The financial news and financial question-and-answer datasets are derived from publicly available stock price series and news on the Internet. We first collect raw stock price series data through the application programming interfaces (APIs) of Tushare and AKShare, and raw social news corpora through the APIs of official media outlets such as CCTV.com, Wall Street Journal, and Eastmoney. We then use large models such as ChatGPT to extract summaries from the raw corpora and provide corresponding prompt words to generate relevant financial questions, which can fully expand the diversity of the dataset samples.

[0049] The financial image analysis dataset comes from publicly available stock price series data on the Internet. First, the stock price series data of all listed companies in the past 100 trading weeks are collected through the Wadi Rabbit interface. Observation indicators such as candlestick charts, Bollinger Bands (BOLL), moving average convergence divergence (MACD), and stochastic indicator (KDJ) are calculated. The indicator images are generated using the PLT drawing tool. Finally, multimodal large models such as the Tongyi Qianwen Visual Understanding Model (Qwen-VL) are used to perform technical analysis on the indicator images and generate corresponding analysis text.

[0050] As an example, the multimodal large language model (financial model) used in this study consists of three components: a visual encoder, a large language model, and an adapter. This multimodal model has visual understanding capabilities and can perform multimodal tasks such as optical character recognition and visual question answering. Common multimodal large models include the Tongyi Qianwen Visual Understanding Model (Qwen-VL) and the Jiang Ziya Multimodal Model (Ziya-Visual).

[0051] The prompts are constructed in text format and include role designation, task instructions, response format constraints, and knowledge document references. For example, the prompt provided to the multimodal financial model is "Please provide a detailed description or analysis of the chart provided to you. The description should include seven aspects: 1. What is the overall trend of the various moving averages on the chart (downward, upward, or sideways)? 2. What is the technical structure of the recent K-line combination and whether there are buy or sell signals? 3. The latest BOLL indicator, whether there are buy or sell signals? 4. The latest MACD indicator, whether there are buy or sell signals? 5. The latest KDJ indicator, whether there are buy or sell signals in terms of trading volume? 6. Is the overall K-line fluctuation range large? 7. And possible future scenarios. The description must be no less than 600 words!"

[0052] (b) Fine-tune the financial model using the constructed financial dataset.

[0053] During fine-tuning of large financial models, the low-rank adaptive LoRA method is used. The existing parameters of the large language model are frozen, and only the parameters of the LoRA module are updated. The LoRA method prevents overfitting of the model to the dataset, which can lead to catastrophic forgetting. It also reduces training memory usage and training costs. Furthermore, batch gradient descent training is performed on randomly selected samples from the multimodal financial dataset.

[0054] As an embodiment, the financial model refers to any pre-trained general-purpose large language model, such as Wenxin Yiyan, Tongyi Qianwen, ChatGLM, etc.

[0055] (c) Use the general large model as the intelligent agent model to identify the intent of user questions and generate corresponding API request parameters or database retrieval parameters.

[0056] Knowledge documents are mainly divided into two categories: online documents and local documents. Online documents can be obtained remotely by requesting an API; local documents are usually stored in a local database and retrieved through local retrieval. In this embodiment, an intelligent agent model is introduced to identify the intent of user questions and generate request parameters. The intelligent agent model can use open source models such as ChatGLM2-6B and Qwen-7B, or API call models such as ChatGPT. By designing detailed and accurate prompt words, the intelligent agent model determines the need to request relevant knowledge documents and what type of knowledge documents to request by answering the current user question, and generates a JSON object of API request or database retrieval parameters in the format constrained by the prompt words:

[0057] (R,P)=Agent(Q)

[0058]

[0059] Among them, R and P represent the intention recognition result set and request parameter set of the intelligent agent model Agent for the user question Q, r j The value is 0 or 1, representing the use of API request or database retrieval, respectively. j The corresponding request parameter. j Represents knowledge documents related to the user's question Q obtained through API requests or database retrieval.

[0060] By designing a detailed and comprehensive set of agent prompts, we achieve intent recognition and parameter generation for the agent model. These prompts are constructed in text form and primarily consist of a role specification ("You are an intelligent AutoGPT. You need to make decisions independently and cannot seek assistance from the user"), a task description, a description of the available tools and their parameters, response format constraints, and user questions. Table 1 lists the prompts provided to the agent.

[0061] Table 1 Prompt words provided to the agent model

[0062]

[0063]

[0064]

[0065] (d) Rewrite the original document obtained. By designing corresponding document rewriting prompts for documents of different modal types, the large model can rewrite the knowledge document. The document rewriting prompts are constructed in text form. For documents with market price time series data, the prompts first describe the stock-related information, then list the time series data of the stock, and finally describe the task, requiring the large model to provide extraction and analysis. For documents with stock research reports, the prompts first provide stock-related information, then provide the research report content, and finally require the large model to perform analysis and summary. For image documents, the prompts require the model to perform detailed analysis and interpretation of different technical indicators based on different parts of the image.

[0066] The coarse-grained original documents obtained in step (c) are typically long and contain a lot of noise, making it difficult for financial models to capture effective information that helps answer user questions. Therefore, large models such as ChatGPT are used to rewrite the original documents separately, extracting the documents and key information to obtain fine-grained entity-level documents.

[0067] d′ j =LLM rewrite (d j )

[0068] Where d′ j Represents the fine-grained document after rewriting by the ChatGPT model. The prompt words used in this step are shown in Table 2.

[0069] Table 2 Prompt words provided to the document rewriting model

[0070]

[0071]

[0072] (e) Re-rank and reverse rank fusion of all acquired knowledge documents

[0073] In the process of generating answers through autoregression of financial models, large models will pay more attention to characters at the front and are more likely to ignore characters at the back. Therefore, the order of knowledge documents in the input also has an important impact on the quality of the large model's response. Therefore, the BM25 algorithm is used for sparse retrieval in the local database, and the inverse sort fusion method is used to merge the results of vector retrieval and sparse retrieval. At the same time, a Cross-Encoder-based method is used to re-rank all knowledge documents obtained from API requests and database retrieval. Compared with the vector retrieval of the Bi-Encoder method, the Cross-Encoder method has a slower calculation speed but higher accuracy. Therefore, it is suitable for re-fine sorting the coarse sorting results of vector retrieval.

[0074] D=Rerank(d′1,d′2,...,d′ j )

[0075] Where D represents the re-ranked knowledge document set.

[0076] (f) Combine knowledge documents with financial task prompts and input them into fine-tuned financial models to conduct effective financial analysis

[0077] Finally, the knowledge document and user questions obtained in step (e) are input into the fine-tuned financial model obtained in step (b) for final financial analysis:

[0078] Res=LLM tuned (D, Q)

[0079] Where D represents the re-sorted knowledge document set, and Q represents the user question. For example, the prompt word Prompt used is "The reference content is as follows:\n <docs>As a financial expert AI, please answer the following questions based on the above references. Fabricated answers are not allowed. \n[User Query]:\n <query>". The comparison of the answer effect of the financial model using this method and the ChatGPT model without this method is shown in Figure 2. Figure 5 shown.

[0080] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, numerous modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention.< / query> < / docs>

Claims

1. An end-to-end agent-based financial model approach suitable for multimodal financial analysis tasks, characterized by: The following steps are involved: (a) Constructing a multimodal financial dataset; (b) Fine-tuning financial models using multimodal financial datasets; (c) Use a general large model as the agent model to identify the intent of the user question and generate corresponding API request parameters or database search parameters; the agent model uses the ChatGLM2-6B, Qwen-7B open source model, or uses the ChatGPT model API call model; by setting prompt words, the agent model determines whether to request relevant knowledge documents and what type of knowledge documents to request by answering the current user question, and generates a JSON object of API request or database search parameters according to the format constrained by the prompt words: (R,P)=Agent(Q) Among them, R and P represent the intention recognition result set and request parameter set of the intelligent agent model Agent for the user question Q, r j The value is 0 or 1, representing the use of API request or database retrieval, respectively. j For r j The corresponding request parameter, d j Represents knowledge documents related to the user's question Q obtained through API requests or database retrieval; (d) Rewrite the original knowledge documents obtained from API requests and knowledge base retrieval; (e) Re-rank and reverse-rank all acquired knowledge documents; (f) Combine knowledge documents with financial task prompts and input them into the fine-tuned financial model to conduct effective financial analysis.

2. The end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks according to claim 1, characterized in that: The multimodal financial dataset includes: a general financial dataset, a financial image analysis dataset, a financial news dataset, and a financial question-answering dataset; Among them, the general financial dataset consists of community open source English financial datasets, which are used to enhance the information extraction and text summarization capabilities of large models; Financial news and financial question-answering datasets are derived from publicly available stock price series and news on the Internet. They are used to expand the diversity of dataset samples and enhance the financial analysis capabilities of large models. The financial image analysis dataset comes from publicly available stock price series data on the Internet and is used to enhance the large model's ability to understand the financial indicator information in the image.

3. The end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks according to claim 1, characterized in that: Fine-tuning is completed using the low-rank adaptive LoRA method. During the fine-tuning process, the original parameters in the financial model will be frozen, and only the parameters in the newly added LoRA module will be updated.

4. The end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks according to claim 1, characterized in that: In step (d), the ChatGPT model is used to rewrite the original documents, extract the documents and key information, and obtain fine-grained entity-level documents: d′ j =LLM rewrite (d j ) Where d′ j Represents the fine-grained document after rewriting by the ChatGPT model, d j Represents knowledge documents related to the user's question Q obtained through API requests or database retrieval.

5. The end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks according to claim 1, characterized in that: Use the BM25 algorithm for sparse search in the local database, and use the inverse sort fusion method to merge the results of vector search and sparse search. At the same time, use the Cross-Encoder method to re-rank all knowledge documents obtained from API requests and database searches: D=Rerank(d′1,d′2,…,d′ j ) Where D represents the re-ranked knowledge document set, d′ j Represents a fine-grained document after rewriting by the ChatGPT model.

6. The end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks according to claim 1, characterized in that: Input the knowledge documents and user questions obtained in step (e) into the fine-tuned financial model for final financial analysis: Res=LLM tuned (D,Q) Among them, D represents the re-ranked knowledge document set, and Q represents the user question.

7. The end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks according to claim 1, characterized in that: The financial model refers to any pre-trained general-purpose large language model.

8. The end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks according to claim 7, characterized in that: The pre-trained general large language model is the Tongyi Qianwen visual understanding model Qwen-VL and the Jiang Ziya-multimodal model.

9. The end-to-end intelligent agent financial model method suitable for multimodal financial analysis tasks according to any one of claims 1 to 8, characterized in that: The prompt words are constructed in a text form, including role designation, task description, user question, reply format constraint, and knowledge document reference section.

Images