Method and system for improving accuracy of large language model on financial query problem based on external retrieval

Abstract

The invention belongs to the technical field of computers, and particularly relates to a method and system for improving the accuracy of a large language model on financial query problems based on external retrieval. The method comprises the steps of 1, summarizing financial problems; 2, training the large language model based on the summary of the financial query problem in the step 1; 3, the complexity of the financial problem is judged; 4, decomposing the calculation formula into a plurality of sub-formulas based on the judgment in the step 3; step 5, for the sub-formulas decomposed in the step 4, utilizing the big language model trained in the step 2 to obtain final answers of the sub-formulas; and step 6, performing result summation on a summation result in the step 5 according to a calculation formula during decomposition, so that the large language model is automatically solved, and the accuracy of the large language model in financial query is improved. The method is used for solving the problem that a large language model needs a large amount of calculation.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a method and a system thereof for improving the accuracy of a large language model in financial query problems based on external retrieval. Background Art

[0002] The development of large models is an important progress in the field of artificial intelligence in recent years. With the increase in hardware computing power and the improvement of algorithms, researchers have developed larger and larger neural network models with more parameters. These large models perform well in tasks such as natural language processing (NLP) and computer vision (CV), providing powerful tools for solving complex problems. The application needs of vertical fields are becoming more and more urgent for customized and personalized solutions. Although traditional general models can handle general tasks, in specific vertical fields, such as healthcare, finance, law, etc., more refined and specialized models are often required to handle complex problems in the field. The data characteristics and task requirements in these fields may be very different from those in general fields, so customized large models are needed to meet the needs.

[0003] Especially in the financial field, it is often necessary to interact with the database, so the large language model in the financial field has high requirements for computing power and the ability to call the database. Although the large language model itself has a certain computing power, it involves complex calculation expressions and needs to be solved by calling the calculator tool. The large language model has a strong generalization ability, but it still cannot answer questions well in special scenarios. In addition, as the amount of data introduced from the database increases, the computing power of the large language model will be greatly weakened. Summary of the invention

[0004] The present invention provides a method and system for improving the accuracy of a large language model in financial query problems based on external retrieval, so as to solve the problem that a large language model needs to perform a large amount of calculation.

[0005] The present invention is achieved through the following technical solutions:

[0006] A method for improving the accuracy of a large language model in financial query problems based on external retrieval, the method comprising the following steps:

[0007] Step 1: Summarize the financial issues;

[0008] Step 2: Train a large language model based on the summary of financial query questions in step 1;

[0009] Step 3: Determine the complexity of the financial issue;

[0010] Step 4: Decompose the calculation formula based on the judgment in step 3 into multiple sub-formulas;

[0011] Step 5: Use the large language model trained in step 2 to obtain the final answer of each sub-formula decomposed in step 4;

[0012] Step 6: Sum the results of step 5 according to the calculation formula used during decomposition, and let the large language model automatically solve the problem, thereby improving the accuracy of the large language model in financial query problems.

[0013] Furthermore, the step 2 specifically includes the following steps:

[0014] Step 2.1: Summarize the three major financial statements and summarize the calculation formulas in the statements, which match the fields in the database;

[0015] Step 2.2: Set the time information to the format of a certain year and month, and set the query information keyword to one of the three major financial statements;

[0016] Step 2.3: Use time information and question information keywords as dimensions and indicators in the three major financial statements as keywords to construct query questions;

[0017] Step 2.4: Use the keywords in the query information to build a solution to the query formula in the large language model answer, and design the format of the calling tool;

[0018] Step 2.4.1: Identify the problem and identify the key information in the problem;

[0019] Step 2.4.2: For period costs, find the formula;

[0020] Step 2.4.3: Query the specific value according to the found formula;

[0021] Step 2.4.4: Fill the queried value back into the formula and calculate to get the final result.

[0022] Step 2.5: Construct a SQL query template with formula information and time information as dimensions, fill the formula information and time information into the SQL template, use the constructed SQL to perform query operations to obtain specific values, and design the format of the calling tool;

[0023] Step 2.6: Fill the queried value back into the formula, design the format of the calculator tool, and calculate the final result data;

[0024] Step 2.7: Add the data in step 2.6 to the end of the data to indicate the end of the data;

[0025] Step 2.8: Add a special prompt to the user's question to enhance the large language model's ability to recognize such questions;

[0026] Step 2.9: Set the learning rate and batch size to adjust the large language model, that is, set the learning rate to 1e-5 and batch size to 4, and use the data constructed in steps 2.5-2.8 to perform SFT fine-tuning;

[0027] Step 2.10: Control the output of the large language model to achieve the function of calling the tool and finally get specific results.

[0028] Furthermore, the step 2.10 specifically includes the following steps:

[0029] Step 2.10.1: Input the user's question into the big language model, and the big language model will construct a data flow to answer the question.

[0030] That is, identifying the key information contained in the user's question and identifying the report information to be queried;

[0031] For report information, the tool name to generate the call and the parameter information is the name of the data to be queried;

[0032] Step 2.10.2: After the large language model detects the call tool, it pauses the output, calls the tool, and performs text recall on the formula information;

[0033] Step 2.10.3: After the large language model finds the formula information, it continues to output and generates the tool name for querying specific values ​​according to the found formula; in the tool for querying the value, fill the parameter information into the SQL template to complete the query;

[0034] Step 2.10.4: After the large language model completes the query operation, it fills the queried value back into the formula and calls the calculator tool to complete the calculation operation;

[0035] Step 2.10.5: The large language model integrates the calculation results with the question and gives the final answer.

[0036] Furthermore, the step 3 specifically includes the following steps:

[0037] Step 3.1: The large language model infers financial questions and performs formula queries after identifying keywords.

[0038] Step 3.2 The large language model performs text recall on the identified information keywords through query formulas, finds the formula used, and determines the number of query contents involved in the formula. If the number is greater than the specified number, it is decomposed.

[0039] Furthermore, the step 4 specifically includes the following steps:

[0040] Step 4.1: Decompose the formula into multiple sub-formulas and sum them up;

[0041] Step 4.2: Construct a prompt for each formula, where the prompt construction needs to be consistent with the output of the large language model when answering financial query questions.

[0042] Furthermore, the step 5 specifically includes the following steps:

[0043] Step 5.1: The large language model queries the value of the decomposed sub-formula according to the corresponding prompt;

[0044] Step 5.2: After the large language model completes the query operation, it fills the queried value back into the formula and calls the calculator tool to complete the calculation operation;

[0045] Step 5.3: The large language model integrates the calculation results with the question and gives the final answer to the sub-formula;

[0046] Step 5.4: Repeat steps 5.1-5.3 to get the final answer of each sub-formula.

[0047] Furthermore, the step 6 specifically includes the following steps:

[0048] Step 6.1 will get the final answer of each sub-formula;

[0049] Step 6.2 constructs the corresponding prompt, which now contains the formula information and the final answer value of the sub-formula;

[0050] Step 6.3 The large language model infers the prompt. At this time, the large language model only needs to fill the final answer value of the sub-formula back into the calculation formula and call the calculator to complete the calculation to answer the financial question.

[0051] Step 6.4 The large language model gives the final calculation result.

[0052] A system for improving the accuracy of a large language model in financial query problems based on external retrieval, the system using the method for improving the accuracy of a large language model in financial query problems based on external retrieval as described above, the system comprising:

[0053] Problem summary module, summarizes financial problems;

[0054] Model training module, which trains a large language model based on the summary of financial query questions;

[0055] Problem analysis module, to judge the complexity of financial problems;

[0056] The formula decomposition module decomposes the calculation formula into multiple sub-formulas based on the judgment of complexity;

[0057] The sub-formula calculation module uses the trained large language model to obtain the final answer of each sub-formula decomposed in step 2;

[0058] The automatic solution module sums the results according to the calculation formula during decomposition, allowing the large language model to automatically solve the problem, thereby improving the accuracy of the large language model in financial query problems.

[0059] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above method is implemented.

[0060] A computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0061] The beneficial effects of the present invention are:

[0062] The present invention sorts out the process of answering financial questions and decomposes a complex problem into sub-problems for solving. Through this invention, a large language model can answer such questions involving accurate data with a very high accuracy rate without the need for additional fine-tuning of the large language model.

[0063] The present invention can also be expanded to allow a large language model to learn solutions to a variety of problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a system flow chart of the present invention.

[0065] Figure 2 It is a schematic diagram of the decomposition method in the present invention.

[0066] Figure 3 It is a system flow chart of the present invention.

[0067] Figure 4 This is the process of large language model processing problems in the present invention. DETAILED DESCRIPTION

[0068] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0069] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0070] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0071] The following is attached to this application specification Figure 1-3 , the technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0072] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0073] Implementation Method 1

[0074] This embodiment provides a method for improving the accuracy of a large language model in financial query problems based on external retrieval, the method comprising the following steps:

[0075] Step 1: Summarize the financial issues;

[0076] Specifically, step 1.1: extract time keywords and question information keywords for a question about the three major financial statements;

[0077] Step 1.3: For the identified question information keywords, perform text recall through query formulas to find the formulas used;

[0078] Query formula means searching for the corresponding formula using the indicator name;

[0079] Step 1.4: Use the identified time keywords and the formula information in step 1.3 to query the corresponding value through SQL;

[0080] Step 1.5: Use the corresponding value found in step 1.4 to fill in the formula found in step 1.3 and calculate to get the final result;

[0081] This step aims to summarize the general solutions to financial problems. Summarizing the general solutions makes it easier for large language models to learn the methods and automate the processing flow. In addition, it can greatly facilitate subsequent related processing.

[0082] Step 2: Train a large language model based on the summary of financial query questions in step 1;

[0083] Step 3: Determine the complexity of the financial issue;

[0084] Step 4: Decompose the calculation formula based on the judgment in step 3 into multiple sub-formulas;

[0085] Step 5: Use the large language model trained in step 2 to obtain the final answer of each sub-formula decomposed in step 4;

[0086] Step 6: Sum the summed results of step 5 according to the calculation formula during decomposition, and let the large language model automatically solve the problem as shown in the specific example given in this embodiment, so as to improve the accuracy of the large language model in financial query problems.

[0087] Furthermore, the step 2 specifically includes the following steps:

[0088] Step 2.1: Summarize the three major financial statements and summarize the calculation formulas in the statements, which match the fields in the database;

[0089] Step 2.2: Set the time information to the format of a certain year and month, and set the query information keyword to one of the three major financial statements;

[0090] Step 2.3: Use time information and question information keywords as dimensions to construct a query question; specifically, construct a query question such as "How much is the monetary funds in January 2017?";

[0091] Step 2.4: Use the keywords in the query information to build a solution to the query formula in the large language model answer, and design the format of the calling tool;

[0092] Step 2.4.1: Identify the problem and identify the key information in the problem, such as the indicator name. In the following example, the key information is (period cost);

[0093] Step 2.4.2: For period costs, find the formula;

[0094] Step 2.4.3: Query the specific value according to the found formula;

[0095] Step 2.4.4: Fill the queried value back into the formula and calculate to get the final result.

[0096] The specific solution to the query formula in building a large language model answer is:

[0097] Thought: According to the question, the key information extracted is the period cost, and the complete formula needs to be found.

[0098] Action: find_formula [period cost]

[0099] Observation: Period expenses = Level 1 reinsurance expenses + Level 1 sales expenses + Level 1 management expenses + Level 1 financial expenses +

[0100] First-level subject exploration fee

[0101] Thought: Based on the formula information found, we need to know the specific value of the formula in March 2018.

[0102] Action: get_values[March 2018,[First-level subject reinsurance expenses, First-level subject sales expenses, First-level subject management expenses, First-level subject finance

[0103] Cost, first-level subject exploration costs]]

[0104] Observation:{'First-level subject reinsurance expenses': 0,'First-level subject sales expenses': 214563.35,'First-level subject management expenses': 756960.7,'First-level

[0105] Subject financial expenses': 534665.95, 'First-level subject exploration expenses': 0}

[0106] Thought: After getting the specific value, you need to substitute it into the formula: Period cost = Level 1 reinsurance cost + Level 1 sales cost + Level 1

[0107] Project management expenses + first-level financial expenses + first-level exploration expenses, and then calculate using a calculator.

[0108] Action:calculate[0+214563.35+756960.7+534665.95+0]

[0109] Observation:1506190.0

[0110] Thought: According to the calculation results, the period cost is 1506190.0, so the period cost for March 2018 is 1506190.0

[0111] Action:Finish[The period cost for March 2018 is 1506190.0]

[0112] Steps 2.5-2.6 are the process of constructing data for fine-tuning the model based on the idea;

[0113] Step 2.5: Construct a SQL query template with formula information and time information as dimensions, fill the formula information and time information into the SQL template, use the constructed SQL to perform query operations to obtain specific values, and design the format of the calling tool;

[0114] Step 2.6: Fill the queried value back into the formula, design the format of the calculator tool, and calculate the final result data;

[0115] Step 2.7: Add the data in step 2.6 to the end of the data to indicate the end of the data;

[0116] Step 2.8: Add a special prompt to the user's question to enhance the large language model's ability to recognize such questions;

[0117] Step 2.9: Set the learning rate and batch size to adjust the large language model, that is, set the learning rate to 1e-5 and batch size to 4, and use the data constructed in steps 2.5-2.8 to perform SFT fine-tuning;

[0118] Step 2.10: Control the output of the large language model to achieve the function of calling the tool and finally get specific results.

[0119] Furthermore, the step 2.10 specifically includes the following steps:

[0120] Step 2.10.1: Input the user's question into the big language model, and the big language model will construct a data flow to answer the question.

[0121] That is, identifying the key information contained in the user's question and identifying the report information to be queried;

[0122] For report information, the tool name to generate the call and the parameter information is the name of the data to be queried;

[0123] Step 2.10.2: After the large language model detects the call tool, it pauses the output, calls the tool, and performs text recall on the formula information;

[0124] Step 2.10.3: After the large language model finds the formula information, it continues to output and generates the tool name for querying specific values ​​based on the found formula. The parameters are time information and formula information. In the tool for querying values, fill the parameter information into the SQL template to complete the query.

[0125] Step 2.10.4: After the large language model completes the query operation, it fills the queried value back into the formula and calls the calculator tool to complete the calculation operation;

[0126] Step 2.10.5: The large language model integrates the calculation results with the question and gives the final answer.

[0127] According to the above steps, the present invention can enable the large language model to learn how to answer financial questions and answers, and can allow the large language model to be applied to finance-related fields. The specific beneficial effects of the invention are as follows:

[0128] This embodiment follows Figure 4 To proceed with the process shown in the figure, we first need to summarize the problems to be solved, and then construct the problem-solving process data according to the general rules summarized, and use this data to fine-tune the instructions of the large language model. At the same time, the output of the large language model is controlled according to the process. After the user asks the large language model a question, the large language model will give an accurate answer according to the established process.

[0129] Furthermore, the step 3 of judging the complexity of the financial problem mainly based on the length of the formula specifically includes the following steps:

[0130] Step 3.1: The large language model infers financial questions and performs formula queries after identifying keywords.

[0131] Step 3.2 The large language model performs text recall on the identified information keywords through query formulas, finds the formula used, and determines the number of query contents involved in the formula. If the number is greater than the specified number, it is decomposed.

[0132] Specifically, for each indicator in a financial query, there is a calculation formula which is a well-known formula in the financial field.

[0133] Query formula means searching for the corresponding formula using the indicator name;

[0134] For example, this is a complex formula. The specific basis for judgment is that if the number of elements participating in the calculation on the right side of the equal sign is greater than 8, then it is a complex formula, otherwise it is not a complex formula.

[0135] Main business cost = main business cost of first-level account + amortized compensation expenditure of first-level account + amortized insurance liability reserve of first-level account + other business cost of first-level account + unexpired liability reserve of first-level account + insurance liability reserve of first-level account + compensation expenditure of first-level account + policy dividend expenditure of first-level account + surrender of first-level account + premium ceded of first-level account + all first-level accounts starting with account code 5

[0136] 5. The method according to claim 1, characterized in that the step 4 specifically comprises the following steps:

[0137] Step 4.1: Decompose the formula into multiple sub-formulas and sum them up;

[0138] Step 4.2: Construct a prompt for each formula, where the prompt construction needs to be consistent with the output of the large language model when answering financial query questions.

[0139] For example, if the formula is too long, break it down: Main business cost = keyword 1 + keyword 2

[0140] Keyword 1 = Level 1 main business cost + Level 1 amortized claims expenditure + Level 1 amortized insurance liability reserve + Level 1 other business cost + Level 1 unexpired liability reserve + Level 1 insurance liability reserve + Level 1 claims expenditure + Level 1 policy dividend expenditure

[0141] The constructed prompt is as follows, that is, the decomposed formula is used as a sub-question for the large model to answer;

[0142] A respondent to a financial question answers the question along the following lines.

[0143] The solution is: first call to extract keywords, match the extracted keywords with formulas. Then query the specific value based on the matched formula. Finally, substitute the obtained value into the formula and calculate.

[0144] When answering questions, please refer to the following format:

[0145] Thought: answer content, Action: the name of the calling tool, Observation: the return result of calling the tool.

[0146] ###The questions that need to be answered are: <human>:What is keyword 1 in February 2018?

[0147] <bot>: Give the answer

[0148] Furthermore, the step 4 specifically includes the following steps:

[0149] Step 4.1: Decompose the formula into multiple sub-formulas and sum them up;

[0150] Step 4.2: Construct a prompt for each formula, where the prompt construction needs to be consistent with the output of the large language model when answering financial query questions.

[0151] Furthermore, the step 5 specifically includes the following steps:

[0152] Step 5.1: The large language model performs reasoning according to the corresponding prompt. At this time, the large language model no longer needs to call the tool to answer financial questions. The first step of the large language model's reasoning for sub-questions is to query the value based on the decomposed sub-formulas.

[0153] Step 5.2: After the large language model completes the query operation, it fills the queried value back into the formula and calls the calculator tool to complete the calculation operation;

[0154] Step 5.3: The large language model integrates the calculation results with the question and gives the final answer to the sub-formula;

[0155] Step 5.4: Repeat steps 5.1-5.3 to get the final answer of each sub-formula.

[0156] For example, Action: calculate[0+0+0+0+0+0+0+0]

[0157] Observation:0

[0158] Thought: According to the calculation results, keyword 1 is 0, so keyword 1 in February 2018 is 0

[0159] Action:Finish[Keyword 1 in February 2018 is 0];

[0160] After the model calls the calculator, it gets a value. Then the big model combines the value with the question and gets a textual description like (the keyword 1 in February 2018 is 0).

[0161] Furthermore, the step 6 specifically includes the following steps:

[0162] Step 6.1 will get the final answer of each sub-formula;

[0163] Step 6.2 constructs the corresponding prompt, which now contains the formula information and the final answer value of the sub-formula;

[0164] Step 6.3 The large language model infers the prompt. At this time, the large language model only needs to fill the final answer value of the sub-formula back into the calculation formula and call the calculator to complete the calculation to answer the financial question.

[0165] Step 6.4 The large language model gives the final calculation result.

[0166] This embodiment follows Figure 1 To proceed with the process shown in the figure, we first need to judge the problem. If it is highly complex, we need to decompose it. After decomposition, we need to construct the corresponding sub-problems according to the problem-solving process, and answer the sub-problems through the generalization ability of the large model. The answer results of the sub-problems are combined with the formulas before decomposition, and the user's question is reconstructed. The large model can infer the new question to get the final result.

[0167] The following is a question about the three major financial statements, the processing flow of the big model and the final answer. It can be seen that according to this method, the big model can correctly answer the questions according to the established method.

[0168] Example 1

[0169] You are a financial question answerer, and your task is to answer the question according to the following ideas. The answer idea is: first call to extract keywords, and match the extracted keywords with formulas. Then query the specific value according to the matched formula. Finally, substitute the obtained value into the formula and calculate.

[0170] When answering questions, please refer to the following format:

[0171] Thought: answer content, Action: the name of the calling tool, Observation: the return result of calling the tool.

[0172] ###The questions that need to be answered are: <human>:What is the main business cost in February 2018?

[0173] <bot>:

[0174] Thought: According to the question, the key information extracted is the main business cost, and we need to find the complete formula.

[0175] Action: find_formula [Main business cost]\nObservation: Main business cost = Main business cost of first-level account + Amortized compensation expenditure of first-level account + Amortized insurance liability reserve of first-level account + Other business costs of first-level account + Withdrawal of unexpired liability reserve of first-level account + Withdrawal of insurance liability reserve of first-level account + Compensation expenditure of first-level account + Policy dividend expenditure of first-level account + Surrender of first-level account + Premium ceded of first-level account + All first-level accounts with account code starting with 5

[0176] The formula is too long, break it down: Main business cost = keyword 1 + keyword 2

[0177] (Answer keyword 1)

[0178] You are a financial question answerer, and your task is to answer the question according to the following ideas. The answer idea is: first call to extract keywords, and match the extracted keywords with formulas. Then query the specific value according to the matched formula. Finally, substitute the obtained value into the formula and calculate.

[0179] When answering questions, please refer to the following format:

[0180] Thought: answer content, Action: the name of the calling tool, Observation: the return result of calling the tool.

[0181] ###The questions that need to be answered are: <human>:What is keyword 1 in February 2018?

[0182] <bot>:

[0183] Thought: According to the question, the key information extracted is keyword 1, and we need to find the complete formula.

[0184] Action: find_formula[keyword 1]

[0185] Observation: Keyword 1 = Level 1 main business cost + Level 1 amortized compensation expenditure + Level 1 amortized insurance liability reserve + Level 1 other business cost + Level 1 unexpired liability reserve + Level 1 insurance liability reserve + Level 1 compensation expenditure + Level 1 policy dividend expenditure

[0186] Thought: Based on the formula information found, we need to know the specific value of the formula in February 2018.

[0187] Action: get_values[February 2018,[First-level account main business cost, First-level account amortized compensation expenditure, First-level account amortized insurance liability reserve, First-level account other business costs, First-level account unexpired liability reserve, First-level account insurance liability reserve, First-level account compensation expenditure, First-level account policy dividend expenditure]]

[0188] Observation:{'First-level account main business cost':0,'First-level account amortized compensation expenditure':0,'First-level account amortized insurance liability reserve':0,'First-level account other business costs':0,'First-level account unexpired liability reserve':0,'First-level account insurance liability reserve':0,'First-level account compensation expenditure':0,'First-level account policy dividend expenditure':0}

[0189] Thought: After getting the specific value, you need to substitute it into the formula: Keyword 1 = first-level account main business cost + first-level account amortized compensation expenditure + first-level account amortized insurance liability reserve + first-level account other business costs + first-level account withdrawal of unexpired liability reserve + first-level account withdrawal of insurance liability reserve + first-level account compensation expenditure + first-level account policy dividend expenditure, and then calculate it through the calculator.

[0190] Action:calculate[0+0+0+0+0+0+0+0]

[0191] Observation:0

[0192] Thought: According to the calculation results, keyword 1 is 0, so keyword 1 in February 2018 is 0

[0193] Action:Finish[Keyword 1 in February 2018 is 0]

[0194] (Answer keyword 2)

[0195] You are a financial question answerer, and your task is to answer the question according to the following ideas. The answer idea is: first call to extract keywords, and match the extracted keywords with formulas. Then query the specific value according to the matched formula. Finally, substitute the obtained value into the formula and calculate.

[0196] When answering questions, please refer to the following format:

[0197] Thought: answer content, Action: the name of the calling tool, Observation: the return result of calling the tool.

[0198] ###The questions that need to be answered are: <human>:What is the keyword 2 in February 2018?

[0199] <bot>:

[0200] Thought: According to the question, the key information extracted is keyword 2, and we need to find the complete formula.

[0201] Action: find_formula[keyword 2]

[0202] Observation: Keyword 2 = Level 1 subject surrender + Level 1 subject ceded premium + all Level 1 subjects starting with subject code 5

[0203] Thought: Based on the formula information found, we need to know the specific value of the formula in February 2018.

[0204] Action: get_values[February 2018, [first-level account refund, first-level account ceded premium, all first-level accounts starting with account code 5]]

[0205] Observation:{'First-level subject refund':0,'First-level subject ceded premium':0,'All first-level subjects starting with subject code 5':0}

[0206] Thought: After getting the specific value, you need to substitute it into the formula: Keyword 2 = Level 1 Account Surrender + Level 1 Account Concession Premium + All Level 1 Accounts starting with Account Code 5, and then calculate it using a calculator.

[0207] Action: calculate[0+0+0]

[0208] Observation:0

[0209] Thought: According to the calculation results, keyword 2 is 0, so keyword 2 in February 2018 is 0

[0210] Action:Finish[Keyword 2 in February 2018 is 0]

[0211] (Answer the final result)

[0212] Action:Finish[The main business cost in February 2018 is 0]

[0213] Example 2

[0214] How much will it cost during March 2018?

[0215] Thought: According to the question, the key information extracted is the period cost, and we need to find the complete formula.

[0216] Action: find_formula [period cost]

[0217] Observation: Period expenses = Level 1 reinsurance expenses + Level 1 sales expenses + Level 1 management expenses + Level 1 financial expenses +

[0218] First-level subject exploration fee

[0219] Thought: Based on the formula information found, we need to know the specific value of the formula in March 2018.

[0220] Action: get_values[March 2018,[First-level subject reinsurance expenses, First-level subject sales expenses, First-level subject management expenses, First-level subject finance

[0221] Cost, first-level subject exploration costs]]

[0222] Observation:{'First-level subject reinsurance expenses': 0,'First-level subject sales expenses': 214563.35,'First-level subject management expenses': 756960.7,'First-level

[0223] Subject financial expenses': 534665.95, 'First-level subject exploration expenses': 0}

[0224] Thought: After getting the specific value, you need to substitute it into the formula: Period cost = Level 1 reinsurance cost + Level 1 sales cost + Level 1

[0225] Project management expenses + first-level financial expenses + first-level exploration expenses, and then calculate using a calculator.

[0226] Action:calculate[0+214563.35+756960.7+534665.95+0]

[0227] Observation:1506190.0

[0228] Thought: According to the calculation results, the period cost is 1506190.0, so the period cost for March 2018 is 1506190.0

[0229] Action:Finish[The period cost for March 2018 is 1506190.0]

[0230] Implementation Method 2

[0231] This embodiment provides a system for improving the accuracy of a large language model in financial query problems based on external retrieval. The system uses the method for improving the accuracy of a large language model in financial query problems based on external retrieval as described in Embodiment 1. The system includes:

[0232] Problem summary module, summarizes financial problems;

[0233] Model training module, which trains a large language model based on the summary of financial query questions;

[0234] Problem analysis module, to judge the complexity of financial problems;

[0235] The formula decomposition module decomposes the calculation formula into multiple sub-formulas based on the judgment of complexity;

[0236] The sub-formula calculation module uses the trained large language model to obtain the final answer of each sub-formula decomposed in step 2;

[0237] The automatic solution module sums the results according to the calculation formula during decomposition, allowing the large language model to automatically solve the problem, thereby improving the accuracy of the large language model in financial query problems.

[0238] Implementation Method 3

[0239] An embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory is used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected via a bus. Specifically, the processor implements any step in the first embodiment above by running the computer program stored in the memory.

[0240] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0241] The memory may include a read-only memory, a flash memory, and a random access memory, and provides instructions and data to the processor. A part or all of the memory may also include a nonvolatile random access memory.

[0242] It should be understood that if the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned implementation method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method implementations when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0243] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest range consistent with the principles and novel features disclosed herein.

[0244] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the implementation method can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method implementation method, which will not be repeated here.

[0245] It should be noted that the methods and detailed examples provided in the above embodiments can be combined with the devices and equipment provided in the embodiments, and references can be made to each other, and no further details will be given.

[0246] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0247] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal equipment and method can be implemented in other ways. For example, the device / equipment implementation described above is only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0248] The above-mentioned embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above-mentioned embodiments, or replace some of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.< / bot> < / human> < / bot> < / human> < / bot> < / human> < / bot> < / human>

Claims

1. A method for improving the accuracy of a large language model in financial query problems based on external retrieval, characterized in that: The method comprises the following steps: Step 1: Summarize the financial issues; Step 2: Train a large language model based on the summary of financial query questions in step 1; Step 3: Determine the complexity of the financial issue; Step 4: Decompose the calculation formula based on the judgment in step 3 into multiple sub-formulas; Step 5: Use the large language model trained in step 2 to obtain the final answer of each sub-formula decomposed in step 4; Step 6: Sum the results of step 5 according to the calculation formula used during decomposition, and let the large language model automatically solve the problem, thereby improving the accuracy of the large language model in financial query problems.

2. The method according to claim 1, characterized in that The step 2 specifically includes the following steps: Step 2.1: Summarize the three major financial statements and summarize the calculation formulas in the statements, which match the fields in the database; Step 2.2: Set the time information to the format of a certain year and month, and set the query information keyword to one of the three major financial statements; Step 2.3: Use time information and question information keywords as dimensions and indicators in the three major financial statements as keywords to construct query questions; Step 2.4: Use the keywords in the query information to build a solution to the query formula in the large language model answer, and design the format of the calling tool; Step 2.4.1: Identify the problem and identify the key information in the problem; Step 2.4.2: For period costs, find the formula; Step 2.4.3: Query the specific value according to the found formula; Step 2.4.4: Fill the queried value back into the formula and calculate to get the final result. Step 2.5: Construct a SQL query template with formula information and time information as dimensions, fill the formula information and time information into the SQL template, use the constructed SQL to perform query operations to obtain specific values, and design the format of the calling tool; Step 2.6: Fill the queried value back into the formula, design the format of the calculator tool, and calculate the final result data; Step 2.7: Add the data in step 2.6 to the end of the data to indicate the end of the data; Step 2.8: Add a special prompt to the user's question to enhance the large language model's ability to recognize such questions; Step 2.9: Set the learning rate and batch size to adjust the large language model, that is, set the learning rate to 1e-5 and batch size to 4, and use the data constructed in steps 2.5-2.8 to perform SFT fine-tuning; Step 2.10: Control the output of the large language model to achieve the function of calling the tool and finally get specific results.

3. The method according to claim 2, characterized in that The step 2.10 specifically includes the following steps: Step 2.10.1: Input the user's question into the big language model, and the big language model will construct a data flow to answer the question. That is, identifying the key information contained in the user's question and identifying the report information to be queried; For report information, the tool name to generate the call and the parameter information is the name of the data to be queried; Step 2.10.2: After the large language model detects the call tool, it pauses the output, calls the tool, and performs text recall on the formula information; Step 2.10.3: After the large language model finds the formula information, it continues to output and generates the tool name for querying specific values ​​according to the found formula; in the tool for querying the value, fill the parameter information into the SQL template to complete the query; Step 2.10.4: After the large language model completes the query operation, it fills the queried value back into the formula and calls the calculator tool to complete the calculation operation; Step 2.10.5: The large language model integrates the calculation results with the question and gives the final answer.

4. The method according to claim 1, characterized in that: The step 3 specifically comprises the following steps: Step 3.1: The large language model infers financial questions and performs formula queries after identifying keywords. Step 3.2 The large language model performs text recall on the identified information keywords through query formulas, finds the formula used, and determines the number of query contents involved in the formula. If the number is greater than the specified number, it is decomposed.

5. The method according to claim 1, characterized in that The step 4 specifically comprises the following steps: Step 4.1: Decompose the formula into multiple sub-formulas and sum them up; Step 4.2: Construct a prompt for each formula, where the prompt construction needs to be consistent with the output of the large language model when answering financial query questions.

6. The method according to claim 1, characterized in that The step 5 specifically comprises the following steps: Step 5.1: The large language model queries the value of the decomposed sub-formula according to the corresponding prompt; Step 5.2: After the large language model completes the query operation, it fills the queried value back into the formula and calls the calculator tool to complete the calculation operation; Step 5.3: The large language model integrates the calculation results with the question and gives the final answer to the sub-formula; Step 5.4: Repeat steps 5.1-5.3 to get the final answer of each sub-formula.

7. The method according to claim 1, characterized in that The step 6 specifically comprises the following steps: Step 6.1 will get the final answer of each sub-formula; Step 6.2 constructs the corresponding prompt, which now contains the formula information and the final answer value of the sub-formula; Step 6.3 The large language model infers the prompt. At this time, the large language model only needs to fill the final answer value of the sub-formula back into the calculation formula and call the calculator to complete the calculation to answer the financial question. Step 6.4 The large language model gives the final calculation result.

8. A system for improving the accuracy of a large language model in financial query problems based on external retrieval, characterized in that: The system uses the method for improving the accuracy of a large language model in financial query questions based on external retrieval as described in any one of claims 1 to 7, and the system includes: Problem summary module, summarizes financial problems; Model training module, which trains a large language model based on the summary of financial query questions; The problem analysis module determines the complexity of financial problems; The formula decomposition module decomposes the calculation formula into multiple sub-formulas based on the judgment of complexity; The sub-formula calculation module uses the trained large language model to obtain the final answer of each sub-formula decomposed in step 2; The automatic solution module sums the results according to the calculation formula during decomposition, allowing the large language model to automatically solve the problem, thereby improving the accuracy of the large language model in financial query problems.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images