Method and system for improving accuracy of large language model on financial query questions based on external retrieval

Abstract

The application belongs to the technical field of computers, and particularly relates to a method for improving the accuracy of a large language model on a financial query problem based on external retrieval and a system thereof. Step 1: summarizing a financial problem; step 2: training a large language model based on the summary of the financial query problem in step 1; step 3: judging the complexity of the financial problem; step 4: decomposing a calculation formula into multiple sub-formulas based on the judgment in step 3; step 5: obtaining the final answers of the sub-formulas decomposed in step 4 by using the large language model trained in step 2; and step 6: summing up the results in step 5 according to the calculation formula in the decomposition, and automatically solving the large language model to improve the accuracy of the large language model on the financial query problem. The application is used to solve the problem that the large language model needs to perform a large amount of calculation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computers, and particularly relates to a method and system for improving the accuracy of a large language model on a financial query problem based on external retrieval. BACKGROUND

[0002] The development of large models is an important progress in the field of artificial intelligence in recent years. With the enhancement of hardware computing power and the improvement of algorithms, researchers have developed neural network models that are larger and have more parameters. These large models perform well in tasks such as natural language processing (NLP) and computer vision (CV), providing powerful tools for solving complex problems. The application requirements of vertical fields are increasingly demanding for customized and personalized solutions. Although traditional general-purpose models can handle general tasks, they often need more refined and specialized models to handle complex problems in specific vertical fields such as healthcare, finance, and law. The data characteristics and task requirements of these fields may differ greatly from general fields, so customized large models are needed to meet the requirements.

[0003] In particular, for the financial field, it is often necessary to interact with a database, so large language models in the financial field have high requirements for computing power and the ability to call databases. Although the large language model itself has certain computing power, it needs to call a calculator tool to solve complex computational expressions. The generalization ability of the large language model is strong, but it still cannot answer questions well in special scenarios. In addition, as the amount of data introduced from the database increases, the computing power of the large language model will be greatly weakened. SUMMARY

[0004] The application provides a method and system for improving the accuracy of a large language model on a financial query problem based on external retrieval, to solve the problem of large language models needing to perform a large amount of calculations.

[0005] The application is implemented by the following technical solutions:

[0006] A method for improving the accuracy of a large language model on a financial query problem based on external retrieval, the method comprising the following steps:

[0007] Step 1: summarize the financial problem;

[0008] Step 2: train the large language model based on the summary of the financial query problem in step 1;

[0009] Step 3: determine the complexity of the financial problem;

[0010] Step 4: Based on the judgment of step 3, the decomposition of the calculation formula is carried out, which is decomposed into multiple sub-formulas;

[0011] Step 5: The sub-formulas decomposed in step 4 are used to obtain the final answer of each sub-formula by using the large language model trained in step 2;

[0012] Step 6: The sum result of step 5 is summed according to the calculation formula in the decomposition, and the large language model is automatically solved to improve the accuracy of the large language model in financial query problems.

[0013] Further, the step 2 specifically includes the following steps:

[0014] Step 2.1: Summarize the financial three big reports, and summarize the calculation formula in the report, which matches the fields in the database;

[0015] Step 2.2: Set the time information to a certain year and month, and set the question information keywords to a certain item in the financial three big reports;

[0016] Step 2.3: Construct a query question with time information and question information keywords as dimensions, and indicators in the financial three big reports as keywords;

[0017] Step 2.4: Construct the problem solving idea of the query formula in the large language model answer and the format of the calling tool with the question information keywords in the query question;

[0018] Step 2.4.1: Identify the key information in the question;

[0019] Step 2.4.2: Find the formula for period expenses;

[0020] Step 2.4.3: Query the specific value according to the formula found;

[0021] Step 2.4.4: Fill the queried value into the formula to calculate the final result.

[0022] Step 2.5: Construct a sql query template with formula information and time information as dimensions, fill the formula information and time information into the sql template, and use the constructed sql to query the specific value, and design the format of the calling tool;

[0023] Step 2.6: Fill the queried value into the formula to design the format of the calculator tool and calculate the final result data;

[0024] Step 2.7: Add an end-of-data indicator at the end of the data of step 2.6;

[0025] Step 2.8: Add special prompts to user questions to enhance the ability of large language model to recognize this type of question;

[0026] Step 2.9: Set learning rate and batch size to adjust the large language model, i.e. set learning rate to 1e-5 and batch size to 4, and use the data constructed in steps 2.5-2.8 to perform SFT fine-tuning;

[0027] Step 2.10: Control the output of the large language model to achieve the function of calling tools and finally get specific results.

[0028] Further, the step 2.10 specifically includes the following steps:

[0029] Step 2.10.1: input the user question into the large language model, and the large language model will construct a data flow to answer the question,

[0030] That is, identify the key information contained in the user question and identify the report information to be queried;

[0031] For report information, generate the tool name to be called, and the parameter information is the data name to be queried;

[0032] Step 2.10.2: After the large language model detects the calling tool, it pauses the output, calls the tool, and performs text recall on the formula information;

[0033] Step 2.10.3: After the large language model queries the formula information, it continues to output, generates the tool name to query the specific value according to the queried formula, and fills in the parameter information into the sql template in the value query tool to complete the query;

[0034] Step 2.10.4: After the large language model completes the query operation, it fills the queried value back into the formula and calls the calculator tool to complete the calculation operation;

[0035] Step 2.10.5: The large language model integrates the calculation result with the question to give the final answer.

[0036] Further, the step 3 specifically includes the following steps:

[0037] Step 3.1: The large language model reasons about financial problems and identifies keywords to query formulas;

[0038] Step 3.2: The large language model performs text recall on the identified information keywords by querying formulas to find the used formulas, and judges the number of contents to be queried in the formulas. If it is greater than a specified number, it is decomposed.

[0039] Further, the step 4 specifically includes the following steps:

[0040] Step 4.1: decompose the formula into a plurality of sub-formulas for summation;

[0041] Step 4.2: construct a prompt for each formula, wherein the construction of the prompt needs to be consistent with the output of the large language model when answering the financial query question.

[0042] Further, the step 5 specifically includes the following steps:

[0043] Step 5.1: the large language model queries the values according to the corresponding prompt and the decomposed sub-formulas;

[0044] Step 5.2: after the large language model completes the query operation, the queried values are filled back into the formula, and a calculator tool is called to complete the calculation operation;

[0045] Step 5.3: the large language model integrates the calculation result with the question to give the final answer of the sub-formula;

[0046] Step 5.4: repeat steps 5.1-5.3 to obtain the final answer of each sub-formula.

[0047] Further, the step 6 specifically includes the following steps:

[0048] Step 6.1 obtains the final answer of each sub-formula;

[0049] Step 6.2 constructs a corresponding prompt, which contains formula information and the final answer value of the sub-formula;

[0050] Step 6.3: the large language model reasons on the prompt, at this time the large language model answers the financial problem only needs to fill the final answer value of the sub-formula into the calculation formula, and calls the calculator to complete the calculation.

[0051] Step 6.4: the large language model gives the final calculation result.

[0052] A system for improving the accuracy of a large language model on financial query questions based on external retrieval, the system uses the method for improving the accuracy of a large language model on financial query questions based on external retrieval as described above, and the system includes:

[0053] a question summary module that summarizes financial questions;

[0054] a model training module that trains a large language model based on the summary of financial query questions;

[0055] a question analysis module that judges the complexity of financial questions;

[0056] A formula decomposition module decomposes the calculation formula based on the complexity judgment, and decomposes the calculation formula into multiple sub-formulas;

[0057] A sub-formula calculation module obtains the final answer of each sub-formula by using the trained large language model for the sub-formulas decomposed in step 2;

[0058] An automatic solving module sums the results according to the calculation formula during decomposition, and automatically solves the large language model to improve the accuracy of the large language model in financial query problems.

[0059] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method as described above when executing the computer program.

[0060] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as described above.

[0061] The beneficial effects of the present application are:

[0062] The present application can decompose a complex problem into sub-problems for solving by analyzing the process of answering financial problems, and through the present application, the large language model can answer such problems involving accurate data and achieve very high accuracy without additional fine-tuning of the large language model.

[0063] The present application can also be extended to enable the large language model to learn multiple problem solving methods. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The system flowchart of the present application.

[0065] Figure 2 The decomposition method schematic diagram in the present application.

[0066] Figure 3 The system flowchart of the present application.

[0067] Figure 4 The process flow of the large language model in the present application. DETAILED DESCRIPTION

[0068] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0069] It is to be understood that the terminology "including", when used in the present specification and in the following claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0070] It is also to be understood that the terminology used in the present specification and the appended claims is for the purpose of describing the particular embodiments only and is not intended to be limiting.

[0071] The above description is made in connection with the preferred embodiments of the application described in the present specification. Figures 1-3 The technical solutions in the embodiments of the present application are described clearly and completely. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0072] In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in other manners different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0073] Embodiment one

[0074] The embodiment provides a method for improving the accuracy of a large language model on a financial query question based on external retrieval, and the method comprises the following steps:

[0075] Step 1: summarizing the financial question;

[0076] Specifically, step 1.1: extracting time keywords and question information keywords for a question about the three financial statements;

[0077] Step 1.3: for the identified question information keywords, performing text recall by querying the formula to query the used formula;

[0078] Query formula, refers to the use of index name search to the corresponding formula;

[0079] Step 1.4: using the identified time keywords and the formula information in step 1.3, query the corresponding value through sql;

[0080] Step 1.5: using the corresponding value queried in step 1.4, backfill to the formula queried in step 1.3, and calculate the final result;

[0081] This step aims to summarize the general solution method of financial problems, and summarize the general solution method to facilitate the large language model to learn this method and realize the automation of processing flow. In addition, it can greatly facilitate subsequent related processing.

[0082] Step 2: training the large language model based on the summary of step 1 financial query problem;

[0083] Step 3: judging the complexity of the financial problem;

[0084] Step 4: based on the judgment of step 3, the decomposition of the calculation formula is carried out, which is decomposed into multiple sub formulas;

[0085] Step 5: using the large language model trained in step 2, the final answer of each sub formula is obtained;

[0086] Step 6: summing up the results of step 5 according to the calculation formula in the decomposition, so that the large language model automatically solves the specific examples given in this embodiment, and realizes the improvement of the accuracy of the large language model in the financial query problem.

[0087] Further, the step 2 specifically includes the following steps:

[0088] Step 2.1: summarize the three financial statements, and summarize the calculation formula of the content in the statement, which matches the fields in the database;

[0089] Step 2.2: set the time information to the form of a certain year and month, and set the question information keywords to a certain item in the three financial statements;

[0090] Step 2.3: construct a query question with time information and question information keywords as dimensions; Specifically, construct a query question such as how much is the monetary funds in January 2017?

[0091] Step 2.4: using the question information keywords in the query question, constructing the problem solving idea of the query formula in the large language model answer, and designing the format of the calling tool;

[0092] Step 2.4.1: Identify the problem and the key information, such as the indicator name. In the following example, the key information is (period expenses);

[0093] Step 2.4.2: Find the formula for period expenses;

[0094] Step 2.4.3: Query the specific values according to the formula found;

[0095] Step 2.4.4: Backfill the values into the formula and calculate the final result.

[0096] The problem-solving approach for answering the query formula in large language models is as follows,

[0097] Thought: According to the question, the key information is period expenses, which needs to be found.

[0098] Action: find_formula[period expenses]

[0099] Observation: Period expenses = primary subject underwriting expenses + primary subject sales expenses + primary subject management expenses + primary subject financial expenses +

[0100] primary subject exploration expenses

[0101] Thought: According to the formula information found, we need to know the specific values in the formula for March 2018.

[0102] Action: get_values[March 2018, [primary subject underwriting expenses, primary subject sales expenses, primary subject management expenses, primary subject financial expenses

[0103] , primary subject exploration expenses]]

[0104] Observation: {'primary subject underwriting expenses': 0, 'primary subject sales expenses': 214563.35, 'primary subject management expenses': 756960.7, 'primary subject financial expenses': 534665.95, 'primary subject exploration expenses': 0}

[0105]

[0106] Thought: After getting the specific values, we need to substitute them into the formula: period expenses = primary subject underwriting expenses + primary subject sales expenses + primary subject management expenses + primary subject financial expenses + primary subject exploration expenses, and then calculate it through the calculator.

[0107]

[0108] ​​Action: calculate [0 + 214563.35 + 756960.7 + 534665.95 + 0]

[0109] Observation: 1506190.0

[0110] Thought: According to the calculation result, the period expense is 1506190.0, so the period expense in March 2018 is 1506190.0

[0111] Action: Finish [The period expense in March 2018 is 1506190.0]

[0112] Steps 2.5-2.6 are the process of constructing the data of the fine-tuning model according to the thought;

[0113] Step 2.5: Construct a sql query template with formula information and time information as dimensions, fill in the formula information and time information into the sql template, use the constructed sql to query the specific value, design the format of the calling tool;

[0114] Step 2.6: Backfill the queried value into the formula, design the format of the calculator tool, and calculate to get the final result data;

[0115] Step 2.7: Add an end-of-data indicator to the data of step 2.6;

[0116] Step 2.8: Add a special prompt to the user's question to enhance the recognition ability of the large language model for this type of question;

[0117] Step 2.9: Set the learning rate and batch size to adjust the large language model, i.e. set the learning rate to 1e-5 and the batch size to 4, and use the data constructed in steps 2.5-2.8 to perform SFT fine-tuning;

[0118] Step 2.10: Control the output of the large language model to achieve the function of calling tools, and finally get the specific result.

[0119] Further, the step 2.10 specifically includes the following steps:

[0120] Step 2.10.1: input the user's question into the large language model, the large language model will construct a data flow to answer the question,

[0121] That is, it recognizes the key information contained in the user's question and identifies the report information to be queried;

[0122] For report information, generate the tool name to be called, and the parameter information is the data name to be queried;

[0123] Step 2.10.2: After the large language model detects the calling tool, it pauses the output, calls the tool, and performs text recall on the formula information;

[0124] Step 2.10.3: After the large language model queries the formula information, it continues to output, produces the tool name for querying specific values based on the queried formula, and the parameters are time information and formula information; in the tool for querying values, the parameter information is filled into the SQL template to complete the query;

[0125] Step 2.10.4: After the large language model completes the query operation, it fills the queried values back into the formula and calls the calculator tool to complete the calculation operation;

[0126] Step 2.10.5: The large language model integrates the calculation result with the question and gives the final answer.

[0127] The present application can enable the large language model to learn the method of answering financial questions according to the above steps, and can apply the large language model to the financial field. The beneficial effects of the present application are as follows:

[0128] This embodiment is performed according to the process shown in Figure 4 First, the problem to be solved needs to be summarized, and the general rules of the summary are used to construct the problem solving process data, which is used for large language model instruction fine-tuning. At the same time, the output of the large language model is controlled according to the process. After the user asks the large language model, the large language model will give accurate results according to the established process.

[0129] Further, the step 3 mainly determines the complexity of the financial problem according to the formula length, which includes the following steps:

[0130] Step 3.1: The large language model reasons the financial problem and identifies the key words to query the formula;

[0131] Step 3.2: The large language model performs text recall on the identified information keywords through formula query, finds the used formula, and judges the number of contents that need to be queried in the formula. If it is greater than the specified number, it is decomposed.

[0132] Specifically, in the financial query, each indicator has a calculation formula which is a public formula in the financial field.

[0133] Querying the formula means searching for the corresponding formula using the indicator name;

[0134] For example, this is a complex formula, and the specific basis for judgment is that the number of elements participating in the calculation on the right side of the equal sign is greater than 8, which is a complex formula, otherwise it is not a complex formula.

[0135] Main business cost = main business cost of first-level subject + first-level subject of amortized claim payment + first-level subject of amortized insurance liability reserve + first-level subject of other business cost + first-level subject of extracted unexpired liability reserve + first-level subject of extracted insurance liability reserve + first-level subject of claim payment + first-level subject of policy dividend payment + first-level subject of refund premium + first-level subject of outgo premium + all first-level subjects with subject code starting with 5

[0136] 5. The method of claim 1, wherein the step 4 specifically comprises the following steps:

[0137] Step 4.1: decompose the formula into a form of sum of multiple sub-formulas;

[0138] Step 4.2: construct a prompt for each formula, where the construction of the prompt needs to be consistent with the output of the large language model when answering the financial query question.

[0139] For example, the formula is too long, and the formula is decomposed: main business cost = keyword 1 + keyword 2

[0140] Keyword 1 = main business cost of first-level subject + first-level subject of amortized claim payment + first-level subject of amortized insurance liability reserve + first-level subject of other business cost + first-level subject of extracted unexpired liability reserve + first-level subject of extracted insurance liability reserve + first-level subject of claim payment + first-level subject of policy dividend payment

[0141] The constructed prompt is as follows, that is, the decomposed formula is taken as a sub-problem to let the large model answer;

[0142] A responder to a financial question answers the question according to the following idea.

[0143] The solution idea is: first, call the keyword extraction, and then match the extracted keywords with the formula. Then, according to the matched formula, query the specific value. Finally, substitute the obtained value into the formula and calculate.

[0144] When answering the question, the following format needs to be referred to:

[0145] Thought: answer content, Action: call the name of the tool, Observation: the return result of the call tool.

[0146] ### The question to be answered is: <human>What is the keyword 1 in February 2018?

[0147] <bot>Provide the answer

[0148] Furthermore, step 4 specifically includes the following steps:

[0149] Step 4.1: Decompose the formula into multiple sub-formulas and sum them up;

[0150] Step 4.2: Construct a prompt for each formula, where the construction of the prompt needs to be consistent with the output of the large language model when answering financial query questions.

[0151] Furthermore, step 5 specifically includes the following steps:

[0152] Step 5.1: The large language model infers according to the corresponding prompt. At this point, the large language model no longer needs to call the tool to answer financial questions. The first step of the large language model inferring sub-questions is to query the value according to the decomposed sub-formulas.

[0153] Step 5.2: After the large language model completes the query operation, it fills the retrieved value back into the formula and calls the calculator tool to complete the calculation operation;

[0154] Step 5.3: The large language model integrates the calculation results with the question and gives the final answer of the sub-formula;

[0155] Step 5.4: Repeat steps 5.1-5.3 to obtain the final answer for each sub-formula.

[0156] For example, Action: calculate[0+0+0+0+0+0+0+0]

[0157] Observation:0

[0158] Thought: According to the calculation results, keyword 1 is 0, therefore keyword 1 in February 2018 is 0.

[0159] Action: Finish [The keyword for February 2018 is 1, which is 0];

[0160] The model calls the calculator and obtains a numerical value. Then the larger model combines the numerical value with the question to obtain a textual description like (the keyword for February 2018 is 0).

[0161] Furthermore, step 6 specifically includes the following steps:

[0162] Step 6.1 will yield the final answers for each sub-formula;

[0163] Step 6.2 Construct the corresponding prompt, which at this point contains formula information and the final answer value of the sub-formula;

[0164] Step 6.3 The large language model reasons about the prompt, at which point the large language model answers the financial question by simply backfilling the final answer value of the sub-formula into the calculation formula and invoking the calculator to complete the calculation.

[0165] Step 6.4 The large language model gives the final calculation result.

[0166] This embodiment proceeds according to the flow as shown in Figure 1 First, the question needs to be judged, and if there is high complexity, it needs to be decomposed. After decomposition, the corresponding sub-problems need to be constructed according to the problem-solving process, and the large model is used to answer the sub-problems through its generalization ability. The answer to the sub-problem is combined with the formula before decomposition to reconstruct the user's question, and the large model reasons about the new question to obtain the final result.

[0167] The following is a question about the three financial statements, the large model processing flow and the final answer. As can be seen, according to this method, the large model can correctly answer the question according to the established method.

[0168] Example 1

[0169] You are an answerer of financial questions, and your task is to answer the question according to the following thought. The answer thought is: first, call the keyword extraction, match the formula according to the extracted keywords. Then, according to the matched formula, query the specific value. Finally, substitute the obtained value into the formula and calculate.

[0170] When answering the question, refer to the following format

[0171] Thought: answer content, Action: name of the tool called, Observation: return result of the tool called.

[0172] ### The question to be answered is: <human>What is the cost of main business in February 2018?

[0173] <bot>:

[0174] Thought: Based on the question, the key information we extracted is the cost of main business operations, and we need to find the complete formula.

[0175] Action: find_formula[Main Business Costs]\nObservation: Main Business Costs = Primary Account Main Business Costs + Primary Account Amortized Claims Expenses + Primary Account Amortized Insurance Liability Reserves + Primary Account Other Business Costs + Primary Account Unearned Premium Reserves + Primary Account Insurance Liability Reserves + Primary Account Claims Expenses + Primary Account Policy Dividend Expenses + Primary Account Surrender Payments + Primary Account Cephalosed Premiums + All Primary Accounts with Codes Starting with 5

[0176] The formula is too long, so we break it down: Main business cost = Keyword 1 + Keyword 2

[0177] (Answer keyword 1)

[0178] You are a respondent to a financial question, and your task is to answer the question using the following approach: First, extract keywords and perform formula matching on the extracted keywords. Then, look up the specific value based on the matched formula. Finally, substitute the obtained value into the formula and calculate.

[0179] When answering questions, please refer to the following format.

[0180] Thought: The response content; Action: The name of the tool being invoked; Observation: The result returned by the tool.

[0181] The questions that need to be answered are: <human>What is the keyword 1 in February 2018?

[0182] <bot>:

[0183] Thought: According to the question, the key information extracted is keyword1, and we need to find the complete formula.

[0184] Action: find_formula[keyword1]

[0185] Observation: keyword1 = Level 1 Subject Main Business Cost + Level 1 Subject Recovered Claim Payment + Level 1 Subject Recovered Insurance Liability Reserve + Level 1 Subject Other Business Cost + Level 1 Subject Extracted Unexpired Liability Reserve + Level 1 Subject Extracted Insurance Liability Reserve + Level 1 Subject Claim Payment + Level 1 Subject Policy Dividend Payment

[0186] Thought: According to the formula information found, we need to know the specific values in the formula for February 2018.

[0187] Action: get_values[February 2018, [Level 1 Subject Main Business Cost, Level 1 Subject Recovered Claim Payment, Level 1 Subject Recovered Insurance Liability Reserve, Level 1 Subject Other Business Cost, Level 1 Subject Extracted Unexpired Liability Reserve, Level 1 Subject Extracted Insurance Liability Reserve, Level 1 Subject Claim Payment, Level 1 Subject Policy Dividend Payment]]

[0188] Observation: {'Level 1 Subject Main Business Cost': 0, 'Level 1 Subject Recovered Claim Payment': 0, 'Level 1 Subject Recovered Insurance Liability Reserve': 0, 'Level 1 Subject Other Business Cost': 0, 'Level 1 Subject Extracted Unexpired Liability Reserve': 0, 'Level 1 Subject Extracted Insurance Liability Reserve': 0, 'Level 1 Subject Claim Payment': 0, 'Level 1 Subject Policy Dividend Payment': 0}

[0189] Thought: After getting the specific values, we need to substitute them into the formula: keyword1 = Level 1 Subject Main Business Cost + Level 1 Subject Recovered Claim Payment + Level 1 Subject Recovered Insurance Liability Reserve + Level 1 Subject Other Business Cost + Level 1 Subject Extracted Unexpired Liability Reserve + Level 1 Subject Extracted Insurance Liability Reserve + Level 1 Subject Claim Payment + Level 1 Subject Policy Dividend Payment, and then calculate it using a calculator.

[0190] Action: calculate[0 + 0 + 0 + 0 + 0 + 0 + 0 + 0]

[0191] Observation: 0

[0192] Thought: According to the calculation result, keyword1 is 0, so keyword1 in February 2018 is 0

[0193] Action: Finish [The keyword for February 2018 is 1, which is 0]

[0194] (Answer Keyword 2)

[0195] You are a respondent to a financial question, and your task is to answer the question using the following approach: First, extract keywords and perform formula matching on the extracted keywords. Then, look up the specific value based on the matched formula. Finally, substitute the obtained value into the formula and calculate.

[0196] When answering questions, please refer to the following format.

[0197] Thought: The response content; Action: The name of the tool being invoked; Observation: The result returned by the tool.

[0198] The questions that need to be answered are: <human>What is the keyword 2 in February 2018?

[0199] <bot>:

[0200] Thought: Based on the question, the key information we extracted is keyword 2, and we need to find the complete formula.

[0201] Action: find_formula [keyword 2]

[0202] Observation: Keyword 2 = Primary Subject Surrender Payment + Primary Subject Prepayment + All Primary Subjects with Subject Codes Starting with 5

[0203] Thought: Based on the formula information we found, we need to know the specific values ​​in the formula for February 2018.

[0204] Action: get_values[February 2018, [Primary Subject Surrender Payment, Primary Subject Premium Allocated, All Primary Subjects with Subject Code Starting with 5]]

[0205] Observation:{'First-level subject surrender value':0,'First-level subject ceded premium':0,'All first-level subjects with subject codes starting with 5':0}

[0206] Thought: After obtaining the specific values, you need to substitute them into the formula: Keyword 2 = Primary Subject Surrender Payment + Primary Subject Deferred Premium + All Primary Subjects with Subject Code Starting with 5, and then calculate using a calculator.

[0207] Action: calculate[0+0+0]

[0208] Observation:0

[0209] Thought: According to the calculation results, keyword 2 is 0, therefore keyword 2 in February 2018 is 0.

[0210] Action: Finish [The keyword for February 2018 is 2, which is 0]

[0211] (Answer the final result)

[0212] Action: Finish [The main operating cost in February 2018 was 0]

[0213] Example 2

[0214] What were the expenses for March 2018?

[0215] Thought: Based on the question, the key information we extracted is the period cost, and we need to find the complete formula.

[0216] Action: find_formula[period cost]

[0217] Observation: Period expenses = Reinsurance expenses (Primary category) + Selling expenses (Primary category) + Administrative expenses (Primary category) + Financial expenses (Primary category)

[0218] Primary subject: exploration costs

[0219] Thought: Based on the formula information we found, we need to know the specific values ​​in the formula from March 2018.

[0220] Action:get_values[March 2018,[Level 1 Reinsurance Expenses, Level 1 Sales Expenses, Level 1 Administrative Expenses, Level 1 Financial Expenses]

[0221] Costs, Level 1 subject: Exploration Costs

[0222] Observation:{'Primary Subject Reinsurance Expenses':0,'Primary Subject Sales Expenses':214563.35,'Primary Subject Administrative Expenses':756960.7, ...

[0223] 'Financial Expenses': 534,665.95, 'First-level Exploration Expenses': 0}

[0224] Thought: After obtaining the specific values, they need to be substituted into the formula: Period Expenses = Primary Category Reinsurance Expenses + Primary Category Sales Expenses + Primary Category

[0225] The project management expenses, the first-level financial expenses, and the first-level exploration expenses are then calculated using a calculator.

[0226] Action:calculate[0+214563.35+756960.7+534665.95+0]

[0227] Observation: 1506190.0

[0228] Thought: According to the calculation, the period expense is 1,506,190.0, therefore, the period expense for March 2018 is 1,506,190.0.

[0229] Action: Finish [The period fee for March 2018 was 1,506,190.0]

[0230] Implementation Method 2

[0231] The embodiment provides a system for improving accuracy of a large language model on a financial query question based on external retrieval, the system uses the method for improving accuracy of the large language model on the financial query question based on the external retrieval in the embodiment one, and the system comprises:

[0232] a question summarizing module configured to summarize the financial question;

[0233] a model training module configured to train the large language model based on the summary of the financial query question;

[0234] a question analysis module configured to determine the complexity of the financial question;

[0235] a formula decomposition module configured to decompose a calculation formula based on the complexity determination into a plurality of sub-formulas;

[0236] a sub-formula calculation module configured to obtain final answers of the sub-formulas decomposed in step 2 by using the trained large language model;

[0237] an automatic solution module configured to sum the results according to the calculation formula in the decomposition, and let the large language model automatically solve the sum to improve the accuracy of the large language model on the financial query question.

[0238] Embodiment three

[0239] The electronic device provided by the embodiment of the present application comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the memory is configured to store software programs and modules, and the processor is configured to execute various functional applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected through a bus. Specifically, the processor realizes any step in the above embodiment one by running the above computer program stored in the memory.

[0240] It should be understood that, in the embodiment of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0241] The memory can include read-only memory, flash memory, and random access memory, and provide the processor with instructions and data. Part or all of the memory can also include non-volatile random access memory.

[0242] It should be understood that the above-mentioned integrated modules / units, if implemented in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiments can also be completed by computer programs instructing related hardware, and the above-mentioned computer programs can be stored in a computer readable storage medium. When the computer programs are executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. The above-mentioned computer programs include computer program codes, which can be in the form of source code, object code, executable files or some intermediate forms. The above-mentioned computer readable medium can include any entity or device capable of carrying the above-mentioned computer program codes, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier wave signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents included in the above-mentioned computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0243] The above description of disclosed embodiments enables those skilled in the art to carry out or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0244] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above-mentioned device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0245] It should be noted that the method and its details provided by the above-mentioned embodiments can be combined into the device and equipment provided by the embodiments, and mutual reference will not be described here.

[0246] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0247] In the embodiments provided by the present application, it should be understood that the disclosed device / terminal equipment and method can be implemented in other ways. For example, the above-mentioned device / device embodiment is only schematic, for example, the division of the above-mentioned modules or units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0248] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.< / bot> < / human> < / bot> < / human> < / bot> < / human> < / bot> < / human>

Claims

1. A method for improving the accuracy of large language models in financial query problems based on external retrieval, characterized in that, The method includes the following steps: Step 1: Summarize the financial issues; Step 2: Train the large language model based on the summary of the financial query problem in Step 1; Step 3: Assess the complexity of the financial issues; Step 4: Based on the judgment in Step 3, decompose the calculation formula into multiple sub-formulas; Step 5: Using the large language model trained in Step 2, the sub-formulas decomposed in Step 4 are used to obtain the final answer of each sub-formula; Step 6: Summing the results from Step 5 according to the calculation formula used during decomposition, allowing the large language model to solve automatically, thereby improving the accuracy of the large language model in financial query problems; Step 2 specifically includes the following steps: Step 2.1: Summarize the three major financial statements and derive the calculation formulas for the contents of the statements. These calculation formulas should be matched with the fields in the database. Step 2.2: Set the time information to a specific year and month, and set the keywords for the question information to a specific item in the three major financial statements; Step 2.3: Construct a query question using time information and query keywords as dimensions, and indicators from the three major financial statements as keywords; Step 2.4: Using keywords from the query information, construct a problem-solving approach for the query formula in the large language model answer, and design the format for calling the tool; Step 2.4.1: Identify the problem and extract key information from it; Step 2.4.2: Find the formula for period expenses; Step 2.4.3: Query the specific value according to the found formula; Step 2.4.4: Fill the retrieved values ​​back into the formula and perform the calculation to obtain the final result; Step 2.5: Construct an SQL query template based on formula information and time information. Fill the SQL template with the formula information and time information, use the constructed SQL to perform a query operation to obtain specific values, and design the format for calling the tool. Step 2.6: Fill the retrieved values ​​back into the formula, design the calculator tool's format, and perform calculations to obtain the final result. Step 2.7: Add a symbol to the end of the data in Step 2.6 to indicate the end of the data; Step 2.8: Add a special prompt to the user's question to enhance the large language model's ability to recognize this type of question; Step 2.9: Adjust the large language model by setting the learning rate and batch size, i.e., set the learning rate to 1e-5 and the batch size to 4, and perform SFT fine-tuning using the data constructed in steps 2.5-2.8; Step 2.10: Control the output of the large language model to enable the use of tools and ultimately obtain the specific results; Step 4 specifically includes the following steps: Step 4.1: Decompose the formula into multiple sub-formulas and sum them up; Step 4.2: Construct a prompt for each formula, where the construction of the prompt needs to be consistent with the output of the large language model when answering financial query questions.

2. The method according to claim 1, characterized in that, Step 2.10 specifically includes the following steps: Step 2.10.1: Input the user's question into the large language model. The large language model will construct a data flow to answer the question. That is, to identify the key information contained in the user's question and to identify the report information to be queried; For report information, specify the name of the tool used to generate the report, and the parameter information is the name of the data to be queried. Step 2.10.2: After the large language model detects the tool call, it pauses the output, calls the tool, and performs text retrieval of the formula information; Step 2.10.3: After the large language model retrieves the formula information, it continues to output the tool name for querying specific values ​​based on the retrieved formula; in the tool for querying values, the parameter information is filled into the SQL template to complete the query; Step 2.10.4: After the large language model completes the query operation, it fills the retrieved value back into the formula and calls the calculator tool to complete the calculation operation; Step 2.10.5: The large language model integrates the calculation results with the question and provides the final answer.

3. The method according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: The large language model reasons about financial problems, identifies keywords, and then performs formula lookups; Step 3.2 The large language model uses a query formula to retrieve the identified information keywords and finds the formula used. At the same time, it judges the number of queries involved in the formula. If the number is greater than the specified number, it is decomposed.

4. The method according to claim 1, characterized in that, Step 5 specifically includes the following steps: Step 5.1: The large language model queries the values ​​based on the decomposed sub-formulas according to the corresponding prompt; Step 5.2: After the large language model completes the query operation, it fills the retrieved value back into the formula and calls the calculator tool to complete the calculation operation; Step 5.3: The large language model integrates the calculation results with the question and gives the final answer of the sub-formula; Step 5.4: Repeat steps 5.1-5.3 to obtain the final answer for each sub-formula.

5. The method according to claim 1, characterized in that, Step 6 specifically includes the following steps: Step 6.1 will yield the final answers for each sub-formula; Step 6.2 Construct the corresponding prompt, which at this point contains formula information and the final answer value of the sub-formula; Step 6.3 The large language model infers from the prompt. At this point, the large language model only needs to fill the final answer value of the sub-formula back into the calculation formula and call the calculator to complete the calculation to answer the financial question. Step 6.4 The large language model gives the final calculation results.

6. A system for improving the accuracy of large language models in financial query problems based on external retrieval, characterized in that, The system uses the method described in any one of claims 1-5 for improving the accuracy of large language models on financial query problems based on external retrieval, and the system includes: The problem summary module summarizes financial issues. The model training module trains the large language model based on a summary of financial query questions; The problem analysis module assesses the complexity of financial issues. The formula decomposition module decomposes the calculation formula into multiple sub-formulas based on the complexity judgment. The sub-formula calculation module uses a trained large language model to obtain the final answer of each sub-formula from the sub-formulas decomposed in step 2. The automatic solution module sums the results according to the calculation formula used during decomposition, allowing the large language model to solve automatically and thus improving the accuracy of the large language model in financial query problems.

7. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-5.

Images