A financial mixed data intelligent question answering method and system based on a large language model

Abstract

The application discloses a kind of financial mixed data intelligent question answering method and system based on large language model.It relates to mixed data intelligent question answering technical field, method includes: step 1: obtain TAT-QA data, and using the data in TAT-QA development set constructs analogy example data set;Step 2: using the analogy example data set constructs analogy example vector library;Step 3: using the data in TAT-QA training set constructs supervised fine-tuning training data set with analogy example;Step 4: using supervised fine-tuning method trains intelligent question answering model;Step 5: construct reasoning pipeline, complete intelligent question answering task.The method of the application can be localized deployment more easily, data use is safer;Reduce the work burden of document analyst and improve its work efficiency.

Description

Technical Field

[0001] This invention relates to the field of hybrid data intelligent question answering technology, and more specifically to a financial hybrid data intelligent question answering method and system based on a large language model. Background Technology

[0002] In today's context of economic and trade globalization, corporate annual reports and other financial documents serve as crucial information carriers of a company's operational status, financial performance, and future development direction. Analyzing these documents is a critical and complex task. This process not only helps uncover potential problems in a company's operations but also provides investors with important decision-making references. Financial documents typically contain a large amount of unstructured text content and structured tabular data: the text content meticulously describes corporate events and operational dynamics using natural language, while the tabular data efficiently presents the mathematical logic of the data through intuitive logical relationships and precise numerical information. Questioning and analyzing these financial documents is not only a key means of revealing operational problems and supporting investment decisions but also an important guarantee for coping with global competition and improving the efficiency of business decision-making.

[0003] In recent years, there has been some research on intelligent question answering for mixed financial data containing text and tables. Generally speaking, previous research can be divided into two main categories: two-stage mixed question answering using specialized retrieval and generator models, and mixed question answering using large language models as input. Specifically, in two-stage mixed question answering, the retrieval unit retrieves relevant evidence from the text and table data, and then the generator answers the question based on the evidence; mixed question answering using large language models relies on the model's existing knowledge to answer the question based on the text and table. Like other natural language processing tasks, mixed question answering using large language models has demonstrated excellent capabilities. However, mixed data in the financial field may have certain sensitivity, and uploading this data to a cloud-deployed large-parameter language model may pose certain security risks. Using a locally deployed, smaller-parameter large language model faces challenges such as limited knowledge reserves and poor understanding of the input content.

[0004] To effectively reduce the workload of financial professionals and significantly improve their work efficiency, it is necessary to develop an intelligent question-answering technology for the financial field that can efficiently process mixed text and table data and achieve intelligent question-answering for professional financial questions.

[0005] Therefore, proposing a financial hybrid data intelligent question answering method and system based on a large language model to solve the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a financial hybrid data intelligent question answering method and system based on a large language model, which can be easily deployed locally, and the data usage is more secure; it reduces the workload of document analysts and improves their work efficiency.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a financial hybrid data intelligent question answering method based on a large language model, comprising:

[0008] Step 1: Obtain TAT-QA data and construct an analog sample dataset using data from the TAT-QA development set;

[0009] Step 2: Construct an analogy sample vector library using the analogy sample dataset;

[0010] Step 3: Construct a supervised fine-tuning training dataset with analog examples using data from the TAT-QA training set;

[0011] Step 4: Train the intelligent question-answering model using supervised fine-tuning methods;

[0012] Step 5: Build the inference pipeline to complete the intelligent question answering task.

[0013] Preferably, constructing the analogy sample dataset includes: extracting data text P, table T, question Q, answer type At, evidence E, expression Eq, answer A, and answer scale S from the development set; filling the data from the development set into a preset instruction template to generate text instructions; inputting the text instructions into a large language model to generate solution process text; and filling the solution process text into a preset analogy sample template to construct the analogy sample dataset.

[0014] Preferably, the preset instruction template includes the task definition and requirements for generating reasoning steps, with key information separated by HTML tags.

[0015] Preferably, the construction of the analogy sample vector library includes: extracting the analogy sample question Q, and obtaining the feature matrix M through a word embedding layer, as shown in the following formula:

[0016] M = Embedding(Q), M ∈ R n×d

[0017] Where d is the feature dimension of the hidden layer of the model;

[0018] The embedded data is used to build the FAISS database, which links and stores the remaining information in the development set with the Q question.

[0019] Preferably, the process of constructing the supervised fine-tuning training dataset includes: extracting text P, table T, question Q, and answer-related metadata from the training set; retrieving the most similar analogy from the analogy sample vector library based on question Q; filling the training data and analogy sample dataset into a preset supervised fine-tuning instruction template to obtain a supervised fine-tuning training dataset with analogy samples.

[0020] Preferably, the supervised fine-tuning instruction template includes three parts: instruction, input, and output, with key information wrapped in HTML tags.

[0021] Preferably, the process of training the intelligent question-answering model includes: taking the text instruction as existing text, predicting each word of the response text, combining the text instruction and the response text into a target text, using Teacher-Forcing technology to transform the text generation task into a word classification task, calculating the classification loss at each time step, and not calculating the loss of the text instruction part.

[0022] Preferably, the process of constructing the reasoning pipeline includes: retrieving the analogy sample vector library based on the question Q in the test set to obtain similar analogy sample data; filling the test data and analogy sample data into a preset instruction template to generate text instructions, wherein the preset instruction template outputs the answer type, evidence, expression, answer and scale in sequence.

[0023] Preferably, a financial hybrid data intelligent question-answering system based on a large language model includes:

[0024] Data acquisition module: Used to acquire TAT-QA data and construct an analog sample dataset using data from the TAT-QA development set;

[0025] Analogy Sample Vector Library Module: Used to construct an analogy sample vector library using the analogy sample dataset;

[0026] Training Dataset Module: Used to construct a supervised fine-tuning training dataset with analog examples using data from the TAT-QA training set;

[0027] Model training module: Used to train intelligent question answering models using supervised fine-tuning methods;

[0028] Inference Pipeline Module: Used to build the inference pipeline to complete intelligent question answering tasks.

[0029] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a financial hybrid data intelligent question answering method and system based on a large language model, the beneficial effects of which are:

[0030] Based on the model and training method of this invention, the best known question-answering performance is achieved on a model with the same number of parameters; the method of this invention can be easily deployed locally, and data use is more secure; the method of this invention can reduce the workload of document analysts in the financial field and improve their work efficiency. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0032] Figure 1 The question-and-answer flowchart provided for this invention;

[0033] Figure 2 The flowchart for training the intelligent question-answering model provided by this invention is shown. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] See Figure 1 As shown in the figure, this invention discloses a financial hybrid data intelligent question answering method based on a large language model, including:

[0036] Step 1: Obtain TAT-QA data and construct an analog sample dataset using data from the TAT-QA development set;

[0037] Step 2: Construct an analogy sample vector library using the analogy sample dataset;

[0038] Step 3: Construct a supervised fine-tuning training dataset with analog examples using data from the TAT-QA training set;

[0039] Step 4: Train the intelligent question-answering model using supervised fine-tuning methods;

[0040] Step 5: Build the inference pipeline to complete the intelligent question answering task.

[0041] Specifically, constructing the analogy sample dataset includes: extracting data text P, table T, question Q, answer type At, evidence E, expression Eq, answer A, and answer scale S from the development set; filling the data from the development set into a preset instruction template to generate text instructions; inputting the text instructions into the large language model to generate solution process text; and filling the solution process text into a preset analogy sample template to construct the analogy sample dataset.

[0042] Furthermore, the TAT-QA dataset includes a development set, a training set, and a validation set, all with consistent characteristics, and can be used as needed at different research stages. That is, the development set and training set data have identical characteristics, differing only in quantity, and are used for different research phases.

[0043] A supervised fine-tuning training dataset with analog examples was constructed using data from the TAT-QA training set, where the analog examples were derived from an analog example vector library.

[0044] Specifically, the preset instruction template includes the task definition and requirements for generating reasoning steps, with key information separated by HTML tags.

[0045] In a specific embodiment of this invention, the hybrid data includes a large amount of text and tables. The text content uses natural language to describe in detail corporate events and operational dynamics, while the tabular data efficiently presents the mathematical logic of the data with intuitive logical relationships and precise numerical information. The research task is to answer financial questions based on the text and tables. Specifically, the input to the intelligent question answering includes text P, table T, and question Q. The model needs to answer the following based on the input: answer type At, evidence E, expression Eq, answer A, and answer scale S. The task definition is shown in the following formula:

[0046] argmax R PT(R|P,T,Q)

[0047] The model response R includes answer type At, evidence E, expression Eq, answer A, and answer scale S.

[0048] Furthermore, the TAT-QA data is acquired, and the data from the TAT-QA development set is used to construct the analogy sample dataset. The specific content is as follows: First, the text P, table T, question Q, answer type At, evidence E, expression Eq, answer A, and answer scale S in the development set are extracted and placed into the preset instruction template for obtaining the reasoning steps to obtain text instructions. Then, the text instructions are input into the large language model, and several solution process texts are generated according to the response format required in the instructions. Finally, the output content is filled into the preset analogy sample template as samples to construct the dataset.

[0049] Specifically, the preset instruction template includes the definition and requirements of the task for generating reasoning steps, especially specifying the answer types, including arithmetic problems, counting problems, multi-segment extraction problems, and single-segment extraction problems. It also notes that key information is separated by HTML tags, which can improve the language model's ability to understand and use input information, resulting in better output response quality.

[0050] In one specific embodiment of the present invention, an analogy sample dataset is constructed by filling a preset analogy sample template with reasoning steps and sample text content as samples. The preset analogy sample template includes sample questions, sample tables, sample text, sample reasoning steps, sample answer types, sample evidence, sample expressions, sample answers, and sample answer scales. Each type of information is separated by HTML tags, which enables the language model to better understand and distinguish information.

[0051] Specifically, the construction of the analogy sample vector library includes: extracting the analogy sample question Q, and obtaining the feature matrix M through a word embedding layer, as shown in the following formula:

[0052] M = Embedding(Q), M ∈ R n×d

[0053] Where d is the feature dimension of the hidden layer of the model;

[0054] The embedded data is used to build the FAISS database, which links and stores the remaining information in the development set with the Q question.

[0055] In a specific embodiment of the present invention, the bge-large-en-v1.5 model is used to perform word embedding on the input question to obtain the feature matrix of the question, which is used as the index of the sample library. The remaining information (text P, table T, answer type At, evidence E, expression Eq, answer A, answer scale S) is used as metadata links for the sample and stored. It is worth mentioning that, in order to make the analogy sample retrieval more accurate, an independent analogy sample vector library is built for each answer type when constructing the FAISS database, that is, four analogy sample vector libraries are finally obtained.

[0056] Specifically, the process of constructing the supervised fine-tuning training dataset includes: extracting text P, table T, question Q, and answer-related metadata from the training set; retrieving the most similar analogy from the analogy sample vector library based on question Q; filling the training data and analogy sample dataset into a preset supervised fine-tuning instruction template to obtain a supervised fine-tuning training dataset with analogy samples.

[0057] Specifically, the supervised fine-tuning instruction template includes three parts: instruction, input, and output, with key information wrapped in HTML tags.

[0058] In a specific embodiment of the present invention, the specific content of constructing a supervised fine-tuning training dataset with analogy examples using data from the TAT-QA training set is as follows: First, obtain the text P, table T, question Q, answer type At, evidence E, expression Eq, answer A, and answer scale S from the training set. Then, based on question Q, retrieve the most similar analogy example (including all metadata) from the analogy example vector library. Place the text P, table T, question Q, analogy example AS, answer type At, evidence E, expression Eq, answer A, and answer scale S into a preset supervised fine-tuning instruction template to obtain a supervised fine-tuning training dataset with analogy examples.

[0059] Specifically, the answer type of the sample is first predicted. Then, word embedding is performed on question Q to obtain its feature matrix. The most similar analogy example is found from the corresponding analogy example vector library, and its metadata is populated into a pre-defined supervised fine-tuning instruction template to construct a supervised fine-tuning training dataset with analogy examples. The pre-defined supervised fine-tuning instruction template includes three parts: instruction, input, and output. It contains a task description, a definition of each question type, a definition of evidence, a definition of expression, a definition of answer, a definition of answer scale, and possible data. All key information is wrapped in HTML tags.

[0060] See Figure 2 As shown, specifically, the process of training the intelligent question-answering model includes: taking the text instruction as existing text, predicting each word of the response text, combining the text instruction and the response text into a target text, using Teacher-Forcing technology to transform the text generation task into a word classification task, calculating the classification loss at each time step, but not calculating the loss of the text instruction part.

[0061] In one specific embodiment of the present invention, since text instructions will always be given in the form of known text in actual use, it is not necessary to calculate the classification loss of the model in the text instruction part during training. Through supervised fine-tuning, the disturbance generation model can learn the generation methods and rules of disturbance items by combining the descriptions in the text instructions during training, thereby enabling the model to have basic disturbance item generation capabilities.

[0062] Specifically, the process of constructing the reasoning pipeline includes: retrieving analogy sample vector library based on question Q in the test set to obtain similar analogy sample data; filling the test data and analogy sample data into a preset instruction template to generate text instructions, wherein the preset instruction template outputs the answer type, evidence, expression, answer and scale in sequence.

[0063] In a specific embodiment of the present invention, the specific content of constructing the inference pipeline to complete the intelligent question answering task of financial hybrid data is as follows: First, obtain text P, table T, and question Q from the test set. Based on question Q, retrieve the most similar analogy (including all metadata) from the analogy sample vector library. Then, place text P, table T, question Q, and analogy sample AS into a preset question answering template to obtain text instructions. Then, input the text instructions into the large language model, requiring the language model to answer the question in the following order: determine the answer type of the question, retrieve relevant evidence, generate a problem-solving expression if necessary, generate the answer, and generate the answer scale. Finally, use relevant evaluation methods to evaluate the generation effect to determine whether it can meet the actual use requirements.

[0064] Specifically, the answer type of the sample is first predicted. Then, word embedding is performed on question Q to obtain its feature matrix. The most similar analogy to the question is found from the corresponding analogy sample vector library, and its metadata is used to fill the preset question-answer template to obtain the text instruction. The preset question-answer template is similar to the preset supervised fine-tuning instruction template, but does not include the output part.

[0065] Specifically, embodiments of the present invention also provide a financial hybrid data intelligent question-answering system based on a large language model, including:

[0066] Data acquisition module: Used to acquire TAT-QA data and construct an analog sample dataset using data from the TAT-QA development set;

[0067] Analogy Sample Vector Library Module: Used to construct an analogy sample vector library using the analogy sample dataset;

[0068] Training Dataset Module: Used to construct a supervised fine-tuning training dataset with analog examples using data from the TAT-QA training set;

[0069] Model training module: Used to train intelligent question answering models using supervised fine-tuning methods;

[0070] Inference Pipeline Module: Used to build the inference pipeline to complete intelligent question answering tasks.

[0071] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0072] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large language model-based financial mixed data intelligent question and answer method, characterized in that, The method comprises the following steps: Step 1: Obtain TAT-QA data, and use the data in the TAT-QA development set to construct an analogy example data set; Step 2: Use the analogy example data set to construct an analogy example vector library; Step 3: Use the data in the TAT-QA training set to construct a supervised fine-tuning training data set with analogy examples; Step 4: Train an intelligent question and answer model using a supervised fine-tuning method; Step 5: Construct a reasoning pipeline to complete the intelligent question and answer task. The construction of the analogy example data set comprises the following steps: extracting data texts P, tables T, questions Q, answer types At, evidences E, expressions Eq, answers A and answer scales S in the development set, filling the data in the development set into a preset instruction template to generate a text instruction, inputting the text instruction into a large language model to generate an answer process text, and filling the answer process text into a preset analogy example template to construct an analogy example data set. The preset instruction template comprises task definitions and requirements for generating reasoning steps, and key information is separated by HTML tags. The construction of the analogy example vector library comprises the following steps: extracting questions Q of analogy examples, obtaining a feature matrix M through a word embedding layer, and the formula is as follows: Wherein, d is the hidden layer feature dimension of the model. An embedded data is used to construct a FAISS database, and the remaining information in the development set is linked with the question Q and stored.

2. The financial mixed data intelligent question answering method based on a large language model according to claim 1, characterized in that, The construction process of the supervised fine-tuning training data set comprises the following steps: extracting texts P, tables T, questions Q and answer related metadata in the training set; retrieving the most similar analogy example from the analogy example vector library according to the question Q; filling the training data and the analogy example data set into a preset supervised fine-tuning instruction template to obtain a supervised fine-tuning training data set with analogy examples.

3. The method according to claim 2, wherein, The preset supervised fine-tuning instruction template comprises three parts of instruction, input and output, and key information is wrapped by HTML tags.

4. The financial mixed data intelligent question answering method based on a large language model according to claim 1, characterized in that, The process of training the intelligent question and answer model comprises the following steps: taking the text instruction as an existing text, predicting each word of the reply text, combining the text instruction and the reply text into a target text, using the Teacher-Forcing method, converting the text generation task into a word classification task, calculating the classification loss of each time step, and not calculating the loss of the text instruction part.

5. The financial mixed data intelligent question answering method based on a large language model according to claim 1, characterized in that, The process of constructing the reasoning pipeline comprises the following steps: retrieving the analogy example vector library according to the question Q in the test set to obtain similar analogy example data; filling the test data and the analogy example data into a preset instruction template to generate a text instruction, and the preset instruction template sequentially outputs the answer type, the evidence, the expression, the answer and the scale.

6. A large language model-based financial mixed data intelligent question answering system, applying the large language model-based financial mixed data intelligent question answering method of any one of claims 1-5, characterized in that, The method comprises the following steps: A data acquisition module is used to obtain TAT-QA data, and use the data in the TAT-QA development set to construct an analogy example data set; An analogy example vector library module is used to use the analogy example data set to construct an analogy example vector library; A training data set module is used to use the data in the TAT-QA training set to construct a supervised fine-tuning training data set with analogy examples; A model training module is used to train an intelligent question and answer model using a supervised fine-tuning method; A reasoning pipeline module is used to construct a reasoning pipeline to complete the intelligent question and answer task.

Images