Large language model prediction and evaluation method for financial statement analysis

Abstract

The invention discloses a large language model prediction and evaluation method for financial statement analysis. The method comprises the following steps: collecting and arranging original financial statement data; anonymization processing and standardization processing are sequentially carried out on the financial statement data; designing a structured thinking chain prompt based on the financial statement data after anonymization processing and standardization processing; inputting the financial statement data subjected to anonymization processing and standardization processing and the thinking chain prompt into a large language model, so that the large language model performs prediction according to the thinking chain prompt; the prediction result output by the large language model is evaluated from multiple dimensions, if the evaluation result of each dimension is within a set error range, an analysis report is generated according to the prediction result, and otherwise, the designed thinking chain prompt is optimized. According to the method, the data processing efficiency is improved, the analysis dimension and depth are improved, and the subjectivity and consistency in analysis are improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of tax audit and generative artificial intelligence, and particularly relates to a large language model prediction and evaluation method for financial statement analysis. Background Art

[0002] Traditional financial statement analysis methods mainly rely on manual experience and fixed financial indicator systems. Analysts need to manually collect data, calculate ratios, and combine personal professional judgments to evaluate the financial status of enterprises. This method is not only time-consuming and laborious, but also easily affected by subjective factors, and the analysis dimensions are relatively limited, making it difficult to respond to market changes in real time. Secondly, traditional financial analysis methods mainly perform linear predictions based on historical data and are difficult to capture the non-linear changes in the market. The most important shortcoming of traditional financial analysis methods lies in cost-effectiveness, which requires high labor costs, training costs, and expansion costs. However, with the introduction of large language model technology, revolutionary changes have taken place in the field of financial analysis. Large language models can quickly process massive amounts of structured and unstructured data, automatically identify key financial indicators and trends, provide multi-dimensional analysis perspectives, and can make predictions based on historical data, accompanied by confidence assessments. Summary of the Invention

[0003] The purpose of the present invention is to provide a large language model prediction and evaluation method for financial statement analysis in view of the deficiencies of the existing technology.

[0004] To achieve the above purpose, the present invention provides a large language model prediction and evaluation method for financial statement analysis, including:

[0005] Collect and organize original financial statement data;

[0006] Anonymize and standardize the financial statement data in sequence;

[0007] Design structured chain-of-thought prompts based on the anonymized and standardized financial statement data;

[0008] Input the anonymized and standardized financial statement data and the chain-of-thought prompts into a large language model so that the large language model makes predictions according to the chain-of-thought prompts;

[0009] Evaluate the prediction results output by the large language model from multiple dimensions. If the evaluation results of each dimension are within the set error range, generate an analysis report according to the prediction results; otherwise, optimize the designed chain-of-thought prompts until the evaluation results of each dimension are within the set error range.

[0010] Further, the financial statements include a balance sheet and an income statement.

[0011] Furthermore, the anonymization process is as follows:

[0012] Delete or replace company-specific information that causes bias, and replace specific fiscal years with relative fiscal years to anonymize the data.

[0013] Furthermore, the standardization process includes:

[0014] Standardize the asset-liability relationship based on the constructed balance sheet balance formula, which is specifically as follows:

[0015] A = L + SE

[0016] where A represents total assets, L represents total liabilities, and SE represents fixed equity;

[0017] Standardize the asset value based on the constructed asset structure standardization formula, which is specifically as follows:

[0018]

[0019] where A normalized represents the asset value after standardization, ranging from 0 - 100%, A i represents the actual value of the i-th asset item, A min represents the minimum observed value of the i-th asset item, A max represents the maximum observed value of the i-th asset item;

[0020] Standardize the debt-to-asset ratio based on the constructed debt structure standardization formula, which is specifically as follows:

[0021]

[0022] where DAR standardized represents the debt-to-asset ratio after standardization, TL represents total liabilities, and TA represents total assets;

[0023] Standardize the revenue based on the constructed revenue standardization formula, which is specifically as follows:

[0024]

[0025] where R normalized represents the revenue after standardization, R j represents the j-th revenue item, R mean represents the average revenue, R std represents the standard deviation of all revenue items.

[0026] Further, the specific structured thought chain includes:

[0027] Requiring the large language model to identify and describe significant changes in each key financial indicator;

[0028] Requiring the large language model to calculate key financial ratios, without restricting the financial ratios to be calculated. However, when calculating financial ratios, the large language model is required to explain the calculation formula and then perform specific calculations;

[0029] Requiring the large language model to give an economic interpretation of each calculated financial ratio;

[0030] Requiring the large language model to predict the company's future profitability, direction of change, and magnitude of change, and give the confidence level for the prediction, and also give specific reasons for the prediction results.

[0031] Further, the evaluation of the prediction results output by the large language model from multiple dimensions includes:

[0032] Comparing the prediction results of the large language model with the prediction results given by analysts during the same period;

[0033] Comparing the prediction results of the large language model with the prediction results of traditional machine learning models;

[0034] Using the BERT model to analyze the analysis report generated by the large language model and evaluate the information content and logic in the analysis report.

[0035] Further, it also includes:

[0036] Regularly using new data to verify the accuracy of the prediction results of the large language model. If the accuracy of the prediction results of the large language model is lower than the set threshold, then optimize the thought chain prompt.

[0037] Further, it also includes:

[0038] Constructing an investment strategy based on the prediction results, comparing each prediction result of the large language model with the actual future data, calculating the Sharpe ratio and alpha return through backtest analysis. If the Sharpe ratio or alpha return is lower than the set minimum threshold, first check whether the volatility of the investment strategy is too high. If the volatility of the investment strategy is not too high, then optimize the thought chain prompt.

[0039] Beneficial effects: Improved data processing efficiency: (1) By relying on the powerful natural language processing capabilities of large language models, the present invention can quickly identify and extract key data from various financial statements, effectively processing both standardized statements and unstructured texts. At the same time, the model can intelligently identify data anomalies, automatically fill in missing values, and continuously learn and optimize during the processing, greatly improving the efficiency and accuracy of data processing, and effectively solving the problems of time-consuming and laborious manual analysis and cumbersome data processing procedures.

[0040] (2) Comprehensive analysis dimensions: Aiming at the limitation of single-dimensional traditional financial analysis, the present invention utilizes the vast pre-trained knowledge of large language models, not only focusing on the calculation of basic financial indicators, but also integrating multi-level factors such as industry competition situation and macroeconomic environment. Through the combination of the large language model in the present invention, this method can automatically associate various relevant information, deeply explore potential correlations, provide a full range of analysis perspectives for decision-makers, and significantly improve the depth and value of analysis.

[0041] (3) Improving subjectivity and consistency in analysis: In response to the problems of strong subjectivity and poor consistency in traditional manual analysis, the present invention proposes a chain of thought analysis framework. By guiding the large language model to imitate the analysis process of financial analysts, it effectively eliminates the interference of personal experience and emotional factors. At the same time, the model always maintains an objective and neutral stance, analyzes in a pure data-driven manner, ensures the consistency and comparability of analysis results, and makes the analysis results more objective, reliable and persuasive.

[0042] (4) Enhanced market adaptability: The present invention can quickly respond to various market changes. Through the deep learning and pattern recognition technologies of large language models, it can identify and process various complex linear relationships, capture the complex interaction relationships between various financial indicators, and at the same time the model also has strong adaptability and can integrate new market information in real time. By introducing the large language model, this method has stronger market adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of the large language model prediction and evaluation method for financial statement analysis according to an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of the chain of thought prompt designed according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The present invention will be further clarified below in conjunction with the accompanying drawings and specific embodiments. These embodiments are implemented on the premise of the technical solution of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention.

[0046] As Figure 1 and Figure 2 shown, an embodiment of the present invention provides a large language model prediction and evaluation method for financial statement analysis, including:

[0047] Collect and organize the original financial statement data. Specifically, the above financial statements include the balance sheet and the income statement. The above financial data includes:

[0048] Anonymize and standardize the financial statement data in sequence. Specifically, the method of the above anonymization is as follows:

[0049] Delete or replace any company-specific information that may cause bias in the model, and replace the specific fiscal year with the relative fiscal year (t represents the current period, t-1 represents the previous period, and so on). For example, 2021 is replaced by t, 2020 is replaced by t-1, and 2019 is replaced by t-2 to anonymize the data. The above company-specific information includes the company name, stock code, and any information that may identify the company.

[0050] Next, use the Compustat Balanced Model to standardize the balance sheet and the income statement to ensure the uniformity of the data format. The method of the above standardization includes:

[0051] Standardize the asset-liability relationship based on the constructed balance sheet balance formula. The specific balance sheet balance formula is as follows:

[0052] A = L + SE

[0053] where A represents total assets, L represents total liabilities, and SE represents fixed equity;

[0054] Standardize the asset value based on the constructed asset structure standardization formula. The specific asset structure standardization formula is as follows:

[0055]

[0056] where A normalized represents the standardized asset value, with a range of 0-100%, A i represents the actual value of the i-th asset item, A min represents the minimum observed value of the i-th asset item, A max represents the maximum observed value of the i-th asset item;

[0057] Standardize the asset-liability ratio based on the constructed liability structure standardization formula. The specific liability structure standardization formula is as follows:

[0058]

[0059] Among them, DAR standardized represents the asset-liability ratio after standardization, TL represents total liabilities, and TA represents total assets;

[0060] The revenue is standardized based on the constructed revenue standardization formula, and the above revenue standardization formula is specifically as follows:

[0061]

[0062] Among them, R normalized represents the revenue after standardization, R j represents the j-th revenue item, R mean represents the average revenue, R std represents the standard deviation of all revenue items.

[0063] The above gives some of the main formulas used in the Compustat balance model in the present invention, and the remaining formulas will not be elaborated. Through the above operations, a standardized and anonymized balance sheet and income statement are obtained. Specifically, the standardized and anonymized balance sheet includes four parts: the time dimension, the asset part, the liability part, and the shareholders' equity part. Among them, the time dimension is the relative fiscal year, such as year t and year t-1; the asset part includes current assets, non-current assets, etc.; the liability part includes current liabilities, non-current liabilities, etc.; the owners' equity includes preferred stock, common stock, minority shareholders' equity, etc. Finally, a balance verification is performed to obtain the total assets. The standardized and anonymized income statement includes five parts: the time dimension, the revenue and gross profit part, the operating profit part, the tax and net profit part, and the earnings per share part. Among them, the time dimension is the relative fiscal year, such as year t and year t-1; the revenue and gross profit part includes net sales, cost of sales, gross profit, etc.; the operating profit includes selling and administrative expenses, operating profit before depreciation, depreciation and amortization, etc.; the tax and net profit part includes profit before tax, current income tax, deferred income tax, net profit (loss), etc.; the earnings per share part includes basic earnings per share and diluted earnings per share.

[0064] Design a structured chain-of-thought prompt based on the anonymized and standardized financial statement data. The above chain-of-thought prompt is used to guide the large language model to analyze according to the thinking of professional analysts, specifically including:

[0065] 1. Require the large language model to identify and describe the significant changes in each key financial indicator.

[0066] 2. Require the large language model to calculate key financial ratios. There is no limit to the financial ratios to be calculated, but when calculating the financial ratios, the large language model is required to explain the calculation formula and then perform specific calculations.

[0067] 3. Require the large language model to give an economic interpretation of each calculated financial ratio.

[0068] 4. Require the large language model to predict the company's future profitability (profit / loss), change direction (increase / decrease), and change amplitude (large / medium / small), and give the confidence level (0 - 100%) for the prediction. Also, give specific reasons for the prediction results.

[0069] Input the anonymized and standardized financial statement data and the chain-of-thought prompts into the large language model so that the large language model makes predictions based on the chain-of-thought prompts.

[0070] Evaluate the prediction results output by the large language model from multiple dimensions. If the evaluation results of each dimension are within the set error range, generate an analysis report based on the prediction results; otherwise, optimize the designed chain-of-thought prompts until the evaluation results of each dimension are within the set error range. Specifically, evaluating the prediction results output by the large language model from multiple dimensions includes:

[0071] 1. Compare the prediction results of the large language model with the prediction results given by analysts during the same period. For example, whether the Mean-Square Error is less than 0.015 and whether the direction accuracy rate is greater than 70%.

[0072] 2. Compare the prediction results of the large language model with the prediction results of traditional machine learning models. For example, whether the observed performance difference is less than 20%.

[0073] 3. Use the BERT model to analyze the analysis report generated by the large language model, and evaluate the information content and logic in the analysis report. For example, score the quality and logical consistency of the output text by BERT (0 - 1 point), observe whether the BERT score is greater than 0.7 and whether the logical consistency score is greater than 0.8.

[0074] The analysis report can include detailed information on quantitative and qualitative analysis. The following is an example of the analysis report, specifically including:

[0075] "Trend Analysis": In the past three years, the company's operating income has shown a steady growth trend, rising from 16199.0 to 26142.0, which reflects the good acceptance of the company's products or services in the market. It is worth noting that the operating costs have also increased significantly, from 4443.0 to 12682.8. If this cost increase trend is not properly controlled, it may put pressure on the company's profitability. However, from a positive perspective, although the growth rate of gross profit has slowed down relatively, it still maintains a growth trend, reflecting that the company has certain advantages in market pricing and cost control.

[0076] "Ratio Analysis": In-depth analysis of the company's financial indicators shows that the operating profit margin in this period (operating profit / net sales = 7065.0 / 26142.0) reaches 27.02%, which means that for every 100 yuan of revenue, 27.02 yuan of operating profit can be generated, reflecting good cost control ability. At the same time, through the result that the asset turnover rate (net sales / total assets = 26142.0 / 346288.0) is 0.08, the asset operation efficiency of the company can be evaluated. Compared with the previous year, the operating profit margin has increased from 25.28% (5391.0 / 21325.0) to 27.02% this year, showing an improvement in operating efficiency; but the decrease in the asset turnover rate indicates a weakening in the asset utilization efficiency. This phenomenon of rising operating profit margin and falling asset turnover rate prompts the management to pay attention to the improvement of asset utilization efficiency while maintaining good profitability.

[0077] "Cause Analysis": The "positive" prediction of earnings per share (EPS) for the next year is mainly based on the observed revenue growth trend and the moderate improvement in the operating profit margin, which reflects the company's good operating expenditure management ability during the sales growth process. However, the decrease in the asset turnover rate and the significant increase in the cost of sales have raised concerns about asset utilization efficiency and cost management, adding uncertainty to the prediction. Therefore, although the company shows the potential to improve profitability, considering that these potential efficiency issues may limit the growth rate of earnings per share, we expect the improvement in earnings per share to remain at a moderate level.

[0078] Specifically, the ways to optimize the thought chain prompts for the design are as follows:

[0079] First, determine the main problem sources through error analysis (such as systematic deviations, random fluctuations, logical omissions, etc.), and then implement optimizations accordingly. For data quality issues, strengthen the data preprocessing link, introduce outlier detection and handling mechanisms, and perform feature engineering; for logical problems, refine the analysis steps, add intermediate verification links, and ensure that each reasoning step is supported by sufficient evidence; for knowledge gaps, supplement the latest industry information and professional knowledge, and add relevant case analyses. Then, take the following optimization measures accordingly: for problems at the data level, strengthen the data preprocessing link, introduce outlier detection and handling mechanisms, and perform feature engineering; for problems at the logical level, further refine the reasoning steps, add intermediate verification links, and at the same time require the model to strengthen the explanation of causal relationships to ensure that each reasoning step is supported by sufficient evidence; for problems at the knowledge level, promptly supplement the latest industry dynamic information and professional knowledge, and introduce more actual cases for support; when implementing specifically, an incremental approach should be adopted. First, solve the problems with the greatest impact, and through the iterative cycle of "optimization - verification - feedback", gradually improve each link of the thought chain. For example, in financial analysis, if it is found that the prediction deviation mainly comes from insufficient consideration of the industry's cyclical characteristics, it is necessary to add an industry cycle analysis module to the thought chain, incorporate seasonal adjustment factors, and verify the adjusted prediction effect in combination with historical data, and continuously optimize until the expected accuracy standard is reached.

[0080] To ensure the effectiveness and adaptability of the analysis framework of the large language model, the accuracy of the large language model's prediction results can be verified regularly using new data. If the accuracy of verifying the large language model's prediction results using new data is above the set threshold, there is no need to optimize the thought chain prompt; otherwise, the thought chain prompt needs to be optimized. Additionally, after obtaining the above prediction results, an investment strategy can be constructed based on the prediction results, including designing an equal - weighted or market - value - weighted investment portfolio. Then, each prediction result of the model will be compared with the actual future data, and through backtesting analysis, indicators such as the Sharpe Ratio and Alpha are observed to evaluate the actual return performance of the strategy.

[0081] The calculation method of Alpha return is as follows:

[0082] Alpha = R p -(R f +β(R m -R f ))

[0083] Wherein, R p represents the actual rate of return of the investment strategy, R f represents the risk - free rate, β represents the beta coefficient of the investment strategy, reflecting its volatility relative to the market benchmark, Rm Represents the return rate of the market benchmark.

[0084] Sharpe ratio S a is calculated as follows:

[0085]

[0086] where R a represents the asset return, R b represents the risk-free rate, E[R a -R b represents the expected excess of the asset return over the benchmark return, σ a represents the excess standard deviation of the asset return, and var is the standard deviation calculation function.

[0087] If the results of the backtest analysis are all within the set threshold range, it indicates that the analysis framework of the large language model has better effectiveness and adaptability, and there is no need to optimize the chain of thought prompts; otherwise, the chain of thought prompts need to be optimized. For the Sharpe ratio and Alpha return in the backtest analysis, a minimum threshold and an ideal threshold are set. Specifically, the threshold of the Sharpe ratio is set to the minimum threshold of 1.0 and the ideal threshold of 1.5. When the Sharpe ratio is lower than the minimum threshold of 1.0, it indicates that the risk-adjusted return of the investment strategy is poor, and the chain of thought prompts may need to be optimized. First, check whether the volatility of the investment strategy is too high. If the volatility of the investment strategy is not too high, then optimize the analysis of risk indicators (such as volatility, leverage ratio) in the chain of thought prompts, and increase the attention to the stability of profitability (such as the volatility of cash flow), etc. The threshold of Alpha return is set to the minimum threshold of 1% and the ideal threshold of 3%. When the Alpha return is lower than the minimum threshold of 1%, it indicates that the investment strategy fails to beat the market benchmark, and the chain of thought prompts may need to be optimized. First, check whether the volatility of the investment strategy is too high. If the volatility of the investment strategy is not too high, then optimize the analysis of the market benchmark and industry comparison in the chain of thought prompts, and increase the attention to the enterprise valuation level (such as price-earnings ratio, price-to-book ratio), etc. If the Sharpe ratio is lower than 1.0 and the Alpha return is lower than 0%, then the chain of thought needs to be comprehensively optimized, including links such as profit prediction, industry comparison, and risk analysis. When the volatility of the investment strategy is too high, the investment strategy should be adjusted first.

[0088] The following is an example of threshold setting:

[0089] Table 1:

[0090] Indicator Minimum Target Ideal Target Optimization Trigger Condition Sharpe Ratio 1.0 1.5 <1.0 Alpha Return 1％ 3％ <0％

[0091] Through a multi-dimensional feedback optimization mechanism, a closed-loop and sustainable optimization financial analysis and prediction system is finally formed.

[0092] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, the parts not specifically described belong to the prior art or common general knowledge. Without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A large language model prediction and evaluation method for financial statement analysis, characterized in that: include: Collect and organize original financial statement data; Anonymizing and standardizing the financial statement data in turn; Design structured thought chain prompts based on anonymized and standardized financial statement data; Inputting the anonymized and standardized financial statement data and thought chain prompts into the large language model, so that the large language model makes predictions according to the thought chain prompts; The prediction results output by the large language model are evaluated from multiple dimensions. If the evaluation results of each dimension are within the set error range, an analysis report is generated based on the prediction results. Otherwise, the designed thinking chain prompts are optimized until the evaluation results of each dimension are within the set error range.

2. A large language model prediction and evaluation method for financial statement analysis according to claim 1, characterized in that: The financial statements include a balance sheet and an income statement.

3. A large language model prediction and evaluation method for financial statement analysis according to claim 2, characterized in that: The anonymization process is as follows: Company-specific information that could lead to bias was removed or replaced, and specific fiscal years were replaced with relative fiscal years to make the data anonymous.

4. The large language model prediction and evaluation method for financial statement analysis according to claim 2, characterized in that: The standardized processing methods include: The asset-liability relationship is standardized based on the constructed balance sheet balance formula, and the balance sheet balance formula is as follows: A=L+SE Among them, A represents total assets, L represents total liabilities, and SE represents fixed equity; The asset value is standardized based on the constructed asset structure standardization formula, and the asset structure standardization formula is as follows: Among them, A normalized Represents the standardized asset value, ranging from 0-100%, A i represents the actual value of the ith asset item, A min represents the minimum observed value of the ith asset item, A max represents the maximum observed value of the i-th asset item; The debt-to-asset ratio is standardized based on the constructed debt structure standardized formula, and the debt structure standardized formula is as follows: Among them, DAR standardized It represents the standardized debt-to-asset ratio, TL represents total liabilities, and TA represents total assets; The income is standardized based on the constructed income standardization formula, which is as follows: Among them, R normalized represents the standardized income, R j represents the jth income item, R mean represents the average income, R std Represents the standard deviation of all income items.

5. The large language model prediction and evaluation method for financial statement analysis according to claim 1, characterized in that: The structured thinking chain specifically includes: The large language model is required to identify and describe significant changes in key financial indicators; The large language model is required to calculate key financial ratios. There is no restriction on the financial ratios to be calculated. However, when calculating financial ratios, the large language model is required to explain the calculation formula and then perform specific calculations; The large language model is required to provide an economic explanation for each calculated financial ratio; The large language model is required to predict the company's future profitability, direction of change, and magnitude of change, and provide confidence in the prediction and specific reasons for the prediction results.

6. The large language model prediction and evaluation method for financial statement analysis according to claim 1, characterized in that: The evaluation of the prediction results output by the large language model from multiple dimensions includes: Compare the predictions of the large language model with those given by analysts during the same period; Compare the predictions of the large language model with those of the traditional machine learning model; The BERT model is used to analyze the analysis report generated by the large language model to evaluate the information content and logic in the analysis report.

7. The large language model prediction and evaluation method for financial statement analysis according to claim 1, characterized in that: Also includes: New data is used regularly to verify the accuracy of the prediction results of the large language model. If the accuracy of the prediction results of the large language model is lower than a set threshold, the thought chain prompt is optimized.

8. The large language model prediction and evaluation method for financial statement analysis according to claim 7, characterized in that: Also includes: Construct an investment strategy based on the prediction results, compare each prediction result of the large language model with the actual future data, calculate the Sharpe ratio and alpha return through backtesting analysis, and if the Sharpe ratio or alpha return is lower than the set minimum threshold, first check whether the volatility of the investment strategy is too high. If the volatility of the investment strategy is not too high, optimize the thinking chain prompts.

Images