Financial mixed data intelligent question and answer method and system based on large language model

Abstract

The invention discloses a financial mixed data intelligent question and answer method and system based on a large language model. The method relates to the technical field of mixed data intelligent question answering and comprises the steps that 1, TAT-QA data is acquired, and an analogy sample data set is constructed by using data in a TAT-QA development set; 2, constructing an analogy sample vector library by using the analogy sample data set; 3, using data in the TAT-QA training set to construct a supervised fine tuning training data set with an analogy sample; 4, training the intelligent question and answer model by using a supervised fine tuning method; and 5, constructing a reasoning pipeline, and completing an intelligent question and answer task. According to the method, localization deployment can be easily carried out, and data use is safer; the workload of document analysts is reduced; and the working efficiency of the document analysts is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of hybrid data intelligent question answering, and more specifically to a financial hybrid data intelligent question answering method and system based on a large language model. Background Art

[0002] In today's globalized economy and trade, analyzing financial documents such as annual reports is a critical and complex task, as they provide crucial information about a company's operations, financial performance, and future development. This process not only helps identify potential operational issues but also provides investors with valuable insights for decision-making. Financial documents typically contain a large amount of unstructured text and structured tabular data. The text uses natural language to meticulously describe corporate events and operational dynamics, while the tabular data efficiently presents the mathematical logic of the data through intuitive logical relationships and precise numerical information. Questioning and analyzing these financial documents is not only a key means of uncovering operational issues and supporting investment decisions, but also a crucial step in addressing global competition and improving the efficiency of business decision-making.

[0003] In recent years, there has been a surge in research on intelligent question answering using mixed financial data containing text and tables. Generally speaking, this work can be categorized into two main categories: two-stage mixed question answering using specialized retriever and generator models, and two-stage mixed question answering using text, tables, and questions as input to a large language model. Specifically, in two-stage mixed question answering, the retriever retrieves relevant evidence from text and tables, and the generator then answers questions based on this evidence. Hybrid question answering using a large language model relies on the model's prior knowledge to answer questions based on the text and tables. Like other natural language processing tasks, mixed question answering using large language models has demonstrated impressive capabilities. However, mixed financial data can be sensitive, and uploading this data to a large, cloud-based language model with large parameters presents security risks. Using a locally deployed, large, and smaller language model with large parameters presents challenges such as limited knowledge and poor understanding of the input content.

[0004] In order to effectively reduce the workload of financial practitioners and significantly improve their work efficiency, it is necessary to develop an intelligent question-answering technology for the financial field that can efficiently process mixed text and table data and realize intelligent question-answering of professional financial issues.

[0005] Therefore, it is an urgent problem for those skilled in the art to propose a financial hybrid data intelligent question answering method and system based on a large language model to solve the difficulties existing in the existing technology. Summary of the Invention

[0006] In view of this, the present invention provides a method and system for intelligent question answering of financial hybrid data based on a large language model, which can be easily deployed locally and makes data use safer; it reduces the workload of document analysts and improves their work efficiency.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for intelligent question answering of financial hybrid data based on a large language model, comprising:

[0008] Step 1: Obtain TAT-QA data and construct an analogy sample dataset using data from the TAT-QA development set;

[0009] Step 2: Using the analogy sample dataset to construct an analogy sample vector library;

[0010] Step 3: Use the data in the TAT-QA training set to construct a supervised fine-tuning training dataset with analogy examples;

[0011] Step 4: Use supervised fine-tuning to train the intelligent question answering model;

[0012] Step 5: Build an inference pipeline to complete the intelligent question answering task.

[0013] Preferably, constructing an analogy sample data set includes: extracting data text P, table T, question Q, answer type At, evidence E, expression Eq, answer A and answer scale S from the development set, filling the data from the development set into a preset instruction template to generate text instructions, inputting the text instructions into a large language model to generate a solution process text; filling the solution process text into a preset analogy sample template to construct an analogy sample data set.

[0014] Preferably, the preset instruction template includes the task definition and requirements for generating reasoning steps, and key information is separated by HTML tags.

[0015] Preferably, the construction of the analogy sample vector library includes: extracting the analogy sample question Q, and obtaining the feature matrix M through the word embedding layer, the formula is as follows:

[0016] M=Embedding(Q),M∈R n×d

[0017] Among them, d is the hidden layer feature dimension of the model;

[0018] The embedded data is used to build the FAISS database, and the remaining information in the development set is linked to question Q and stored.

[0019] Preferably, the supervised fine-tuning training data set construction process includes: extracting text P, table T, question Q and answer-related metadata in the training set; retrieving the most similar analogy sample from the analogy sample vector library according to question Q, filling the training data and the analogy sample data set into a preset supervised fine-tuning instruction template to obtain a supervised fine-tuning training data set with analogy samples.

[0020] Preferably, the supervised fine-tuning instruction template includes three parts: instruction, input, and output, and key information is wrapped with HTML tags.

[0021] Preferably, the process of training the intelligent question-answering model includes: taking the text instructions as existing text, predicting each word of the reply text, combining the text instructions and the reply text into a target text using Teacher-Forcing technology, converting the text generation task into a word classification task, calculating the classification loss at each time step, and not calculating the loss of the text instruction part.

[0022] Preferably, the process of constructing the reasoning pipeline includes: retrieving the analogy sample vector library according to the question Q in the test set to obtain similar analogy sample data; filling the test data and analogy sample data into a preset instruction template to generate text instructions, and the preset instruction template outputs the answer type, evidence, expression, answer and scale in sequence.

[0023] Preferably, a financial hybrid data intelligent question answering system based on a large language model includes:

[0024] Data acquisition module: used to obtain TAT-QA data and build analog sample datasets using data in the TAT-QA development set;

[0025] Analogy sample vector library module: used to construct an analogy sample vector library using the analogy sample data set;

[0026] Training dataset module: used to construct a supervised fine-tuning training dataset with analogy examples using the data in the TAT-QA training set;

[0027] Model training module: used to train intelligent question answering models using supervised fine-tuning methods;

[0028] Inference pipeline module: used to build an inference pipeline to complete intelligent question-answering tasks.

[0029] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a method and system for intelligent question answering of financial hybrid data based on a large language model, which has the following beneficial effects:

[0030] Based on the model and training method of the present invention, the best known question-answering effect is achieved on models with the same number of parameters; the method of the present invention can be relatively easily deployed locally, and data use is safer; the method of the present invention can reduce the workload of document analysts in the financial field and improve their work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0032] Figure 1 A question-and-answer flow chart provided by the present invention;

[0033] Figure 2 This is a flowchart for training the intelligent question-answering model provided by the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] See also Figure 1 As shown, the embodiment of the present invention discloses a financial mixed data intelligent question answering method based on a large language model, comprising:

[0036] Step 1: Obtain TAT-QA data and construct an analogy sample dataset using data from the TAT-QA development set;

[0037] Step 2: Using the analogy sample dataset to construct an analogy sample vector library;

[0038] Step 3: Use the data in the TAT-QA training set to construct a supervised fine-tuning training dataset with analogy examples;

[0039] Step 4: Use supervised fine-tuning to train the intelligent question answering model;

[0040] Step 5: Build an inference pipeline to complete the intelligent question answering task.

[0041] Specifically, constructing an analogy sample dataset includes: extracting the data text P, table T, question Q, answer type At, evidence E, expression Eq, answer A, and answer scale S from the development set, filling the data from the development set into a preset instruction template to generate text instructions, inputting the text instructions into a large language model to generate the answer process text; filling the answer process text into the preset analogy sample template to construct an analogy sample dataset.

[0042] Furthermore, the TAT-QA dataset includes a development set, a training set, and a validation set, all with uniform characteristics, which can be used at different research stages as needed. Specifically, the development set and training set data have the same characteristics, differing only in quantity and being used at different research stages.

[0043] The data in the TAT-QA training set is used to construct a supervised fine-tuning training dataset with analogy examples, where the analogy examples come from the analogy example vector library.

[0044] Specifically, the preset instruction template includes the task definition and requirements for generating reasoning steps, and key information is separated by HTML tags.

[0045] In a specific embodiment of the present invention, hybrid data includes a large amount of text and tables. The text content uses natural language to describe business events and operational dynamics in detail, while the tabular data efficiently presents the mathematical logic of the data with intuitive logical relationships and precise numerical information. The research task is to answer financial questions based on the text and tables. Specifically, the input of intelligent question answering includes text P, table T, and question Q. The model is required to answer the following based on the input: answer type At, evidence E, expression Eq, answer A, and answer scale S. The task definition is shown in the following formula:

[0046] argmax R PT(R|P,T,Q)

[0047] The model response R includes the answer type At, evidence E, expression Eq, answer A, and answer scale S.

[0048] Furthermore, we obtained TAT-QA data and used the data in the TAT-QA development set to construct an analogy sample dataset. The specific content is as follows: first, the text P, table T, question Q, answer type At, evidence E, expression Eq, answer A, and answer scale S in the development set are extracted and placed in the preset instruction template for obtaining the reasoning step to obtain text instructions. Then, the text instructions are input into the large language model, and several answer process texts are generated according to the response format required in the instructions. Finally, the output content is filled into the preset analogy sample template as a sample to construct the dataset.

[0049] Specifically, the preset instruction template includes the definition and requirements of the task of generating reasoning steps, especially specifying the answer types, including arithmetic problems, counting problems, multi-fragment extraction problems, and single-fragment extraction problems, and noting that key information is separated by HTML tags. This can improve the language model's ability to understand and use input information, making the output response quality better.

[0050] In one embodiment of the present invention, an analogy example dataset is constructed by filling a preset analogy example template with reasoning steps and sample text as samples. The preset analogy example template includes a sample question, a sample table, sample text, sample reasoning steps, sample answer types, sample evidence, sample expressions, sample answers, and sample answer scales. Each type of information is separated by HTML tags, which enables the language model to better understand and distinguish information.

[0051] Specifically, the construction of the analogy example vector library includes: extracting the analogy example question Q, and obtaining the feature matrix M through the word embedding layer. The formula is as follows:

[0052] M=Embedding(Q),M∈R n×d

[0053] Among them, d is the hidden layer feature dimension of the model;

[0054] The embedded data is used to build the FAISS database, and the remaining information in the development set is linked to question Q and stored.

[0055] In one specific embodiment of the present invention, the bge-large-en-v1.5 model is used to embed the input question into a word, generating a feature matrix for the question. This matrix serves as the index for the sample library. The remaining information (text P, table T, answer type At, evidence E, expression Eq, answer A, and answer scale S) is linked and stored as metadata for the sample. It is worth mentioning that to ensure more accurate analogy retrieval, a separate analogy vector library is constructed for each answer type when constructing the FAISS database, resulting in four analogy vector libraries.

[0056] Specifically, the supervised fine-tuning training dataset construction process includes: extracting text P, table T, question Q and answer-related metadata in the training set; retrieving the most similar analogy sample from the analogy sample vector library according to question Q, filling the training data and the analogy sample dataset into the preset supervised fine-tuning instruction template to obtain a supervised fine-tuning training dataset with analogy samples.

[0057] Specifically, the supervised fine-tuning instruction template includes three parts: instruction, input, and output, and key information is wrapped in HTML tags.

[0058] In a specific embodiment of the present invention, the data in the TAT-QA training set is used to construct a supervised fine-tuning training dataset with analogy examples. The specific content is: first, the text P, table T, question Q, answer type At, evidence E, expression Eq, answer A, and answer scale S in the training set are obtained, and then the analogy example (including all metadata) most similar to the question Q is retrieved from the analogy example vector library according to the question Q, and the text P, table T, question Q, analogy example AS, answer type At, evidence E, expression Eq, answer A, and answer scale S are placed in a preset supervised fine-tuning instruction template to obtain a supervised fine-tuning training dataset with analogy examples.

[0059] Specifically, we first predict the answer type of the sample, then embed the question Q into a word to obtain its feature matrix, find the analogy example that is most similar to the question from the corresponding analogy example vector library, and fill its metadata into the preset supervised fine-tuning instruction template to construct a supervised fine-tuning training dataset with analogy examples. The preset supervised fine-tuning instruction template consists of three parts: instruction part, input part, and output part. It contains task description, definition of each type of question, evidence definition, expression definition, answer definition, answer scale definition, and possible data. All key information is wrapped in HTML tags.

[0060] See also Figure 2 As shown, specifically, the process of training the intelligent question-answering model includes: taking the text instruction as the existing text, predicting each word of the reply text, combining the text instruction and the reply text into a target text, using the Teacher-Forcing technology, converting the text generation task into a word classification task, calculating the classification loss at each time step, and not calculating the loss of the text instruction part.

[0061] In one embodiment of the present invention, since text instructions are always given in the form of known text in actual use, there is no need to calculate the model's classification loss for the text instructions during training. Supervised fine-tuning allows the distractor generation model to incorporate the instructions in the text instructions during training, initially learning the methods and patterns for generating distractors, thereby enabling the model to acquire basic distractor generation capabilities.

[0062] Specifically, the process of building an inference pipeline includes: searching the analogy sample vector library based on question Q in the test set to obtain similar analogy sample data; filling the test data and analogy sample data into a preset instruction template to generate text instructions. The preset instruction template outputs the answer type, evidence, expression, answer and scale in sequence.

[0063] In a specific embodiment of the present invention, the specific content of constructing an inference pipeline to complete the intelligent question-answering task of financial hybrid data is as follows: first, obtain the text P, table T, and question Q in the test set, and retrieve the analogy example (including all metadata) most similar to the question from the analogy example vector library based on the question Q. Then, place the text P, table T, question Q, and analogy example AS in a preset question-answering template to obtain text instructions, and then input the text instructions into the large language model, requiring the language model to answer the question in the following order: determine the answer type of the question, retrieve relevant evidence, generate a problem-solving expression if necessary, generate an answer, and generate an answer scale; finally, use relevant evaluation methods to evaluate the generation effect to determine whether it can meet actual usage requirements.

[0064] Specifically, the answer type is predicted for the sample. Then, word embedding is performed on the question Q to obtain its feature matrix. The most similar analogy to the question is found from the corresponding analogy vector library. Its metadata is then inserted into the preset question-and-answer template to generate a text instruction. The preset question-and-answer template is similar to the preset supervised fine-tuning instruction template, but does not include the output portion.

[0065] Specifically, an embodiment of the present invention further provides a financial hybrid data intelligent question-answering system based on a large language model, comprising:

[0066] Data acquisition module: used to obtain TAT-QA data and build analog sample datasets using data in the TAT-QA development set;

[0067] Analogy sample vector library module: used to construct an analogy sample vector library using the analogy sample data set;

[0068] Training dataset module: used to construct a supervised fine-tuning training dataset with analogy examples using the data in the TAT-QA training set;

[0069] Model training module: used to train intelligent question answering models using supervised fine-tuning methods;

[0070] Inference pipeline module: used to build an inference pipeline to complete intelligent question-answering tasks.

[0071] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0072] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A financial mixed data intelligent question answering method based on a large language model, characterized by: include: Step 1: Obtain TAT-QA data and construct an analogy sample dataset using data from the TAT-QA development set; Step 2: Using the analogy sample dataset to construct an analogy sample vector library; Step 3: Use the data in the TAT-QA training set to construct a supervised fine-tuning training dataset with analogy examples; Step 4: Use supervised fine-tuning to train the intelligent question answering model; Step 5: Build an inference pipeline to complete the intelligent question answering task.

2. The method for intelligent question answering of financial mixed data based on a large language model according to claim 1, characterized in that: Constructing an analogy sample dataset includes: extracting data text P, table T, question Q, answer type At, evidence E, expression Eq, answer A, and answer scale S from the development set, filling the data from the development set into a preset instruction template to generate text instructions, inputting the text instructions into a large language model to generate a text of the answer process; and filling the text of the answer process into the preset analogy sample template to construct an analogy sample dataset.

3. The method for intelligent question answering of financial mixed data based on a large language model according to claim 2, characterized in that: The preset instruction template includes the task definition and requirements for generating reasoning steps, and key information is separated by HTML tags.

4. The method for intelligent question answering of financial mixed data based on a large language model according to claim 1, characterized in that: The construction of the analogy example vector library includes: extracting the analogy example question Q, and obtaining the feature matrix M through the word embedding layer, the formula is as follows: M=Embedding(Q),M∈R n×d Among them, d is the hidden layer feature dimension of the model; The embedded data is used to build the FAISS database, and the remaining information in the development set is linked to question Q and stored.

5. The method for intelligent question answering of financial mixed data based on a large language model according to claim 2, characterized in that: The supervised fine-tuning training dataset construction process includes: extracting text P, table T, question Q and answer-related metadata in the training set; retrieving the most similar analogy sample from the analogy sample vector library according to question Q, filling the training data and the analogy sample dataset into a preset supervised fine-tuning instruction template, and obtaining a supervised fine-tuning training dataset with analogy samples.

6. The method for intelligent question answering of financial mixed data based on a large language model according to claim 5, characterized in that: The supervised fine-tuning instruction template includes three parts: instruction, input, and output, and key information is wrapped in HTML tags.

7. The method for intelligent question answering of financial mixed data based on a large language model according to claim 2, characterized in that: The process of training the intelligent question-answering model includes: treating the text instruction as existing text, predicting each word of the reply text, combining the text instruction and the reply text into a target text using the Teacher-Forcing method, converting the text generation task into a word classification task, calculating the classification loss at each time step, and not calculating the loss of the text instruction part.

8. The method for intelligent question answering of financial mixed data based on a large language model according to claim 1, characterized in that: The process of building an inference pipeline includes: searching the analogy sample vector library based on question Q in the test set to obtain similar analogy sample data; filling the test data and analogy sample data into a preset instruction template to generate text instructions. The preset instruction template outputs the answer type, evidence, expression, answer and scale in sequence.

9. A financial hybrid data intelligent question answering system based on a large language model, characterized by: include: Data acquisition module: used to obtain TAT-QA data and build analog sample datasets using data in the TAT-QA development set; Analogy sample vector library module: used to construct an analogy sample vector library using the analogy sample data set; Training dataset module: used to construct a supervised fine-tuning training dataset with analogy examples using the data in the TAT-QA training set; Model training module: used to train intelligent question answering models using supervised fine-tuning methods; Inference pipeline module: used to build an inference pipeline to complete intelligent question-answering tasks.

Images