A financial research report structuring identification method and system based on a multi-modal large model

Abstract

The application discloses a financial research report structured identification method and system based on a multi-modal large model, relates to the technical field of text image analysis, and first constructs a training data set by acquiring and labeling financial research report images, and then performs two-stage training on a multi-modal large model based on the data set: first, in the cold start supervised fine-tuning stage, the base model is fine-tuned by using thought chain information, so that the base model learns to follow the collaborative reasoning logic to generate structured output; then, in the reinforcement learning optimization stage, the fine-tuned model is used as an initial strategy, a grouping relative strategy optimization algorithm is adopted, and strategy optimization is performed based on a multi-task reward function. Finally, the trained model is input into the to-be-identified research report image, and the structured result containing a chart list, a text and a thought chain reasoning chain can be output. By introducing the thought chain to guide the model training, the structured degree, accuracy and interpretability of the financial research report information extraction are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text image analysis, in particular, to a financial research report structured identification method and system based on a multi-modal large model. BACKGROUND

[0002] In the field of enterprise financial analysis, investment decision and strategic planning, broker industry research reports and enterprise annual reports are the core information carriers. Realizing the automatic, structured analysis and understanding of such research reports has become a key requirement to improve the intelligent level of financial technology. This process needs to accurately locate, semantically associate, content extract and deeply understand the multi-element heterogeneous information such as text, tables and graphics in the research report.

[0003] However, the current technical solutions applied in this field have obvious limitations and deficiencies, mainly in the following aspects: first, the adaptability of general multi-modal large models in specific financial research report analysis scenarios is insufficient. The core design goal of such models is to interact with graphics and text in general fields, and they are not optimized for the layout structure and semantic features of financial research reports. Therefore, they are difficult to accurately distinguish the semantic boundaries between graphics, tables and text, and lack the ability to identify the specific location of the above semantic elements in the document. This directly leads to the omission or incorrect collection of key data in the information extraction process, which cannot meet the high requirements of financial analysis for data accuracy. Second, single-task models designed specifically for document analysis have fragmented functions and lack systematic understanding capabilities. These models can usually only complete single tasks such as layout detection, table structure identification or text recognition independently, and lack effective information coordination and cross-validation mechanisms between tasks. This leads to the inability of the model to perform deep association understanding and logical consistency verification on the extracted data, especially lacking the ability to judge the "unreadable" situation of low-quality images and ambiguous content. When encountering ambiguous images, such models often still produce unreliable outputs or directly miss them, and their erroneous results are easy to mislead subsequent decision-making. Third, the processing flow constructed by multiple independent models (e.g., layout detection → table identification → OCR text recognition → manual verification) has poor robustness and low efficiency. This solution needs to frequently adapt and align the output formats of different models, which is prone to error accumulation and transmission; at the same time, the deployment of multiple models occupies a large amount of computing resources, resulting in high memory requirements and high operating costs; the strong dependence on manual verification at the end of the process makes it difficult for the entire system to meet the efficient and real-time automatic analysis requirements, becoming a bottleneck for business scaling.

[0004] In summary, the prior art has significant shortcomings in the positioning accuracy of figures / tables / text elements, the collaborative understanding and verification between multi-modal information, the efficiency of end-to-end processing flow, and the reliability of rejecting low-quality inputs, and cannot achieve efficient, accurate and reliable structured understanding of enterprise research reports. SUMMARY

[0005] The purpose of the present application is to overcome the existing technical defects and provide a financial research report structured identification method and system based on a multi-modal large model, which solves the technical problems of low accuracy, poor logical consistency and weak interpretability in financial research report structured identification by introducing a two-stage training technique of thought chain reasoning chain for the multi-modal large model.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] In a first aspect, the present application provides a financial research report structured identification method based on a multi-modal large model, comprising:

[0008] Obtain multiple financial research report images, and label each financial research report image to generate a training data set, the labeling including image identification, scene information, table list, text, and thought chain reasoning chain;

[0009] Based on the training data set, the multi-modal large model is trained in two stages;

[0010] In the cold start supervised fine-tuning stage, the base multi-modal large model is supervised fine-tuned using the thought chain information in the labeled data, so that the base multi-modal large model learns to follow the collaborative reasoning logic defined by the thought chain information, and generates a structured output;

[0011] In the reinforcement learning optimization stage, the fine-tuned model is used as the initial strategy, a reinforcement learning algorithm based on grouping relative strategy optimization is adopted, and the model is optimized based on a set of multi-task reward functions;

[0012] Input the financial research report image to be identified into the trained model to obtain the output structured identification result, which includes the table list, the text, and the thought chain reasoning chain.

[0013] In one possible implementation, the generation of the thought chain reasoning chain comprises:

[0014] Use a layout analysis model to position the semantic structure of the financial research report image to obtain the chart type and the boundary box coordinates;

[0015] Use a multi-modal recognition model to analyze the positioned semantic structure to generate the title and content details of the chart;

[0016] Optical character recognition is performed on the text region of the financial research report image to generate text of the text region;

[0017] The chart type and the bounding box coordinates, the title and the content details of the chart, and the text of the text region are integrated, and manual checking is assisted to generate a thought chain reasoning chain.

[0018] In a possible implementation, the base multimodal large model is supervisedly fine-tuned using the thought chain information in the labeled data, so that the base multimodal large model learns to follow the collaborative reasoning logic defined by the thought chain information to generate a structured output, and the step includes:

[0019] The base multimodal large model is fine-tuned using a parameter-efficient fine-tuning method, wherein the parameter-efficient fine-tuning method is LoRA, and LoRA adapters are added to the visual tower and the language model head of the base multimodal large model;

[0020] The training is performed using a prompt template containing a thought chain reasoning chain, wherein the prompt template includes a user input part and an assistant response part, the user input part contains an image and a task description, the task description requires output of structured information in JSON format and a thought chain reasoning chain, and the assistant response part contains the thought chain reasoning chain and the JSON output;

[0021] The loss function used in the training is a cross-entropy loss function to maximize the conditional probability of the output sequence and the true label.

[0022] In a possible implementation, the multi-task reward function is a weighted sum of a positioning reward, an abstract reward, an OCR reward, a consistency reward, and an integrity reward.

[0023] The positioning reward is calculated based on the intersection over union of the bounding box predicted by the model and the true bounding box;

[0024] The abstract reward is calculated based on the BLEU score of the chart content details generated by the model and the true content details;

[0025] The OCR reward is calculated based on the character error rate of the text of the text region recognized by the model and the true text of the text region;

[0026] The consistency reward is used to verify the logical consistency of the thought chain reasoning chain output by the model and the structured output content;

[0027] The integrity reward is calculated based on the accuracy of the unreadable state judgment of the chart region by the model.

[0028] In a possible implementation, the consistency reward includes a first sub-reward when the positioning logic matches the bounding box, a second sub-reward when the abstract extraction basis matches the content details, and a third sub-reward when the unreadable judgment reason matches the integrity reason.

[0029] The integrity reward includes: the highest reward is given when the model correctly labels the unreadable area and the reason is reasonable, zero reward is given when the model misjudges the readability, and a medium reward is given when the model correctly labels a partially readable area.

[0030] In a possible implementation, the fine-tuned model is used as an initial strategy, a reinforcement learning algorithm based on grouping relative strategy optimization is adopted, and the steps of strategy optimization of the model based on a set of multi-task reward functions include:

[0031] For each training sample, a plurality of candidate outputs are generated by sampling from the current model strategy;

[0032] The total reward of each candidate output under the multi-task reward function is calculated;

[0033] Based on the total reward of all candidate outputs, the relative advantage within each group is calculated;

[0034] The model parameters are optimized by a policy gradient update algorithm guided by the relative advantage, and the KL divergence is used as a constraint term in the optimization process.

[0035] In a possible implementation, the integrity state includes three states of clear, partially readable and unreadable;

[0036] For the chart area judged as unreadable, a preset unreadable identifier is output in the structured identification result, and the reason for unreadable is provided in the thought chain reasoning chain.

[0037] In a second aspect, the present application provides a financial research report structured identification system based on a multi-modal large model, which includes:

[0038] A data construction module is configured to obtain multiple financial research report images and perform labeling to generate a training data set, and the labeling includes image identification, scene information, chart list, text, and thought chain reasoning chain.

[0039] A model training module is configured to perform two-stage training on the multi-modal large model based on the training data set, which includes:

[0040] A cold start fine-tuning unit is configured to perform supervised fine-tuning on the base model using thought chain information;

[0041] A reinforcement learning optimization unit is configured to use the fine-tuned model as an initial strategy, adopt a grouping relative strategy optimization algorithm, and perform strategy optimization of the model based on a multi-task reward function;

[0042] A structured identification module is configured to input a to-be-identified financial research report image into the trained model and obtain a structured identification result including a chart list, text, and a thought chain reasoning chain.

[0043] The above main scheme of the present application and each further selected scheme can be freely combined to form multiple schemes, which are all the schemes that can be adopted and claimed by the present application; and the present application can also be freely combined between each non-conflicting selection and between and other selections. Those skilled in the art can understand that there are many combinations based on the prior art and common knowledge after understanding the schemes of the present application, which are all the technical schemes claimed by the present application, and are not listed here.

[0044] The present application discloses a financial research report structured identification method and system based on a multi-modal large model. First, a training data set is constructed by acquiring and labeling financial research report images, and then the multi-modal large model is trained in two stages based on the data set: first, in the cold start supervised fine-tuning stage, the base model is fine-tuned using the thinking chain information, so that it learns to follow the collaborative reasoning logic to generate structured output; then in the reinforcement learning optimization stage, the fine-tuned model is used as the initial strategy, the grouping relative strategy optimization algorithm is used, and the policy is optimized based on the multi-task reward function. Finally, the to-be-identified research report image is input into the trained model, and the structured result containing the chart list, the text of the main text and the thinking chain reasoning chain can be output. By introducing the thinking chain to guide the model training, the structured degree, accuracy and explainability of the financial research report information extraction are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical schemes of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0046] Figure 1 A flowchart of a financial research report structured identification method based on a multi-modal large model according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] The embodiments of the present application will be described below through specific and concrete examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure. The present application can also be implemented or applied by different specific embodiments, and each detail in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0048] Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0049] To solve the comprehensive problems of existing multi-modal large models and related technical solutions in the structured understanding task of financial research reports and the like documents, an embodiment of the present application proposes a financial research report structured identification method and system based on a multi-modal large model, which can realize chart pixel-level positioning, content accurate summary, high-fidelity OCR of the text, and explainable rejection of unreadable areas, while simplifying the process, improving efficiency and robustness.

[0050] Please refer to Figure 1 , Figure 1 The present application embodiment shows a flowchart of a financial research report structured identification method based on a multi-modal large model, which includes:

[0051] Step S1, obtain multiple financial research report images, and label each financial research report image to generate a training data set, the labeling including image identification, scene information, chart list, text of the main text, and thought chain reasoning chain.

[0052] The chart list includes at least one chart item, each chart item containing chart type, boundary box coordinates, title, content details, integrity status and integrity reason, and the thought chain reasoning chain includes chart positioning logic, chart abstract extraction basis, text OCR verification logic and integrity judgment logic.

[0053] Chart positioning identifies tables (upper left corner, 15% of the layout), line charts (middle, 20% of the layout), and pie charts (right side, 10% of the layout) through layout analysis, and excludes text interference areas. Chart abstract: extract key numerical values (2024 forecast revenue) from tables, extract trend peaks from line charts, and mark unreadable due to the inability to extract from pie charts. Text OCR: associate chart content to verify OCR correctness (e.g., “949.5 million vehicles” is consistent with the annual cumulative value of the line chart, with no contradictions). Integrity judgment: pie chart meets unreadable conditions due to blurring, so reject its abstract and only output “unreadable information”.

[0054] Firstly, multiple financial research report images as original data are needed to be obtained. Then, each financial research report image is finely labeled, and each data sample in the formed training data set contains multiple dimensions of labeling information. Image identification and scene information are used to uniquely identify the image sample and record its macro background. Chart list is used to systematically record the structured information of all chart elements in the research report image. The list contains at least one chart item, and each chart item is an independent data unit. The text of the main text is obtained by optical character recognition or other text extraction techniques on the main text area of the research report image. The thought chain reasoning chain is used to explicitly depict the logical reasoning process that should be followed when analyzing the research report image. The reasoning chain specifically includes: chart positioning logic, chart abstract extraction basis, main text OCR verification logic, and integrity judgment logic.

[0055] The generation step of the thought chain reasoning chain includes:

[0056] The layout analysis model is used to position the semantic structure of the financial research report image to obtain the chart type and boundary box coordinates.

[0057] The multi-modal recognition model is used to analyze the positioned semantic structure to generate the title and content details of the chart.

[0058] Optical character recognition is performed on the main text area of the financial research report image to generate the main text.

[0059] Integrate the chart type and boundary box coordinates, the title and content details of the chart, and the main text, and supplement with manual verification to generate the thought chain reasoning chain.

[0060] The pre-trained layout analysis model is used to preliminarily process the input financial research report image. The model is responsible for deep semantic structure positioning of the image, automatically identifying and segmenting different element regions in the image. For the identified chart region, the model will output its specific chart type and accurate boundary box coordinates in the image.

[0061] Then, for each chart semantic structure positioned in the previous step, the multi-modal recognition model is used for deep analysis to understand the visual information conveyed by the chart image and convert it into structured text description. The model will generate the title of the chart and refine the abstract of the content details of the data trend and core points displayed by the chart. In this process, the model will also generate preliminary reasoning basis. Optical character recognition is performed on the main text area of the financial research report image to convert the text content in image form into machine-readable and processable pure text, i.e. the main text. The OCR engine will output the recognized text content and its position information in the original text.

[0062] Finally, the integration and manual verification phase is entered. In this phase, all intermediate results (chart type, bounding box coordinates, chart title and content details, and OCR text of the body) obtained by the automation steps are summarized. Based on this information, a labeling expert manually writes, organizes, and finally forms a complete thought chain reasoning chain. Finally, all information is integrated to assess the readability of the chart and form the final integrity judgment logic.

[0063] Step S2, based on the training data set, two-stage training is performed on the multi-modal large model;

[0064] Step S21, in the cold start supervised fine-tuning stage, the base multi-modal large model is supervised fine-tuned using the thought chain information in the labeled data, so that the base multi-modal large model learns to follow the collaborative reasoning logic defined by the thought chain information to generate structured output;

[0065] Step S22, in the reinforcement learning optimization stage, the fine-tuned model is used as the initial strategy, a reinforcement learning algorithm based on grouping relative strategy optimization is adopted, and the model is optimized based on a set of multi-task reward functions;

[0066] The cold start supervised fine-tuning stage is to let the model initially learn the defined collaborative reasoning logic. The training data set, especially the thought chain reasoning chain contained therein, is used to supervise the fine-tuning of the base multi-modal large model. In this process, the training target of the model is forced to learn and reproduce the complete thought chain in the labeled data. By letting the model generate a complete sequence containing intermediate reasoning steps and final structured results in a self-recurrent manner after being given an input image, the model is guided to understand and internalize the complete analysis path from perception to reasoning to output defined by the thought chain information.

[0067] After the cold start fine-tuning establishes the correct reasoning mode for the model, the supervised fine-tuned model is used as the initial strategy, and a reinforcement learning algorithm based on grouping relative strategy optimization is used to deeply optimize the model. The core of this algorithm is: for the same training sample, a plurality of candidate outputs are generated from the current model strategy to form a group; then, the total reward of each candidate output is calculated based on the multi-task reward function. This set of reward functions is a comprehensive evaluation system that includes positioning reward, summary reward, OCR reward, consistency reward, and integrity reward, etc. in multiple dimensions, which can accurately depict the core needs in the financial research report structured recognition task. The optimization process is not directly pursuing the maximization of the absolute value of the reward, but based on the relative advantage of all candidate outputs in the group, the model parameters are adjusted through the strategy gradient update algorithm.

[0068] The multi-task reward function is the weighted sum of the positioning reward, the summary reward, the OCR reward, the consistency reward, and the integrity reward.

[0069] The positioning reward is calculated based on the intersection over union of the bounding box predicted by the model and the real bounding box;

[0070] The abstract reward is calculated based on the BLEU score of the chart content details generated by the model and the real content details;

[0071] The OCR reward is calculated based on the character error rate of the text recognized by the model and the real text;

[0072] The consistency reward is used to verify the logical consistency between the thought chain reasoning chain output by the model and the structured output content;

[0073] The integrity reward is calculated based on the accuracy of the model's judgment of the unreadable state of the chart area.

[0074] The positioning reward is calculated based on the intersection over union of the bounding box predicted by the model and the real bounding box to improve the accuracy of chart positioning; the abstract reward is evaluated based on the BLEU score between the chart content details generated by the model and the real content details, aiming to optimize the quality of abstract generation; the OCR reward is evaluated by calculating the character error rate of the text recognized by the model and the real text, to improve the accuracy of text recognition. The consistency reward is specifically used to verify the logical consistency between the thought chain reasoning chain output by the model and the structured output content, to ensure that the reasoning process and the final result are mutually confirmed. The integrity reward is calculated based on the accuracy of the model's judgment of the unreadable state of the chart area, to enhance the robustness of the model in processing low-quality images.

[0075] In one possible implementation, the multi-task reward function is designed as:

[0076] Total reward = positioning reward (Loc_R) + abstract reward (Cap_R) + OCR reward (Ocr_R) + reasoning-output consistency reward (Con_R) + integrity judgment reward (Int_R).

[0077] The rewards are defined as follows:

[0078] Positioning reward (Loc_R): based on the IOU gradient calculation of bbox, IOU≥0.95→1.0; 0.8≤IOU<0.95→0.6; 0.6≤IOU<0.8→0.3; IOU<0.6→0, distinguishing between "accurate positioning", "approximate positioning", and "wrong positioning";

[0079] Abstract reward (Cap_R): calculate BELU by scene, encourage accurate abstract and correct rejection;

[0080] Ocr_R: Calculate the CER character error rate of main_txt and GT to ensure the accuracy and coherence of OCR;

[0081] Con_R: Verify the logical consistency of CoT reasoning and output: the positioning logic matches the bbox → 0.3; the summary basis matches the content_detail → 0.4; the unreadable reason matches the integrity_reason → 0.3; none of the above → 0;

[0082] Int_R: Improve the reliability of the answer and the accuracy of the unreadable area judgment: correctly label "unreadable" with reasonable reasons → 1.0; misjudgment (clearly label as unreadable / vice versa) → 0; partially readable label is correct → 0.6.

[0083] The consistency reward includes: when the positioning logic matches the bounding box, a first sub-reward is given, when the summary extraction basis matches the content detail, a second sub-reward is given, and when the unreadable judgment reason matches the integrity reason, a third sub-reward is given;

[0084] The integrity reward includes: when the model correctly labels the unreadable area and the reason is reasonable, the highest reward is given, when the model misjudges the readability, zero reward is given, and when the model correctly labels the partially readable area, a medium reward is given.

[0085] When the positioning logic described in the thought chain and the actual predicted bounding box can be mutually verified in space, the first sub-reward is given; when the summary extraction basis stated in the reasoning chain and the content detail generated in the end are semantically consistent, the second sub-reward is given; when the judgment reason for the readability of the chart is consistent with the integrity reason logic in the annotation data, the third sub-reward is given.

[0086] When the model can correctly label the unreadable area and provide reasonable reasons, the highest reward will be obtained to encourage the model to make reliable and conservative decisions when facing low-quality images; when the model misjudges the readability, zero reward is given to punish such key errors that may mislead downstream tasks; when the model correctly labels the partially readable area, a medium reward is given to recognize its accurate processing ability for complex and boundary conditions.

[0087] Step S3, input the financial research report image to be recognized into the trained model to obtain the output structured recognition result.

[0088] The user inputs a financial research report image to be identified into a trained multi-modal large model. The model performs end-to-end automatic analysis on the input image based on the collaborative reasoning logic learned in the previous stage. The structured identification result output by the model is a machine-readable and processable data set containing multi-dimensional information. The result not only contains the key content directly extracted from the research report, i.e., the chart list and the text of the main text, but also generates a complete thought chain reasoning chain.

[0089] The base multi-modal large model is supervised fine-tuned using the thought chain information in the labeled data, so that the base multi-modal large model learns to follow the collaborative reasoning logic defined by the thought chain information to generate structured output. The steps include:

[0090] The base multi-modal large model is fine-tuned using a parameter-efficient fine-tuning method, which is LoRA, and LoRA adapters are added to the visual tower and language model head of the base multi-modal large model.

[0091] The training is performed using a prompt template containing a thought chain reasoning chain, the prompt template including a user input part and an assistant response part, the user input part containing an image and a task description, the task description requiring output of structured information in JSON format and a thought chain reasoning chain, and the assistant response part containing a thought chain reasoning chain and a JSON output.

[0092] The loss function used in training is a cross-entropy loss function to maximize the conditional probability of the output sequence and the true label.

[0093] The base multi-modal large model is fine-tuned using a parameter-efficient fine-tuning method, which is LoRA. LoRA adapters are mainly added to the visual tower and language model head of the base model, which can efficiently fine-tune the key modules of the model for processing multi-modal information. In the organization of training data, a prompt template containing a thought chain reasoning chain is used. The template is strictly divided into a user input part and an assistant response part. The user input part contains a financial research report image to be processed and a clear task description, which requires the model to output JSON format structured data containing chart list, main text, etc. information, and requires a complete thought chain reasoning process. The assistant response part is the supervised learning target of the model, which contains the standard thought chain reasoning chain provided in the labeled data and the JSON output corresponding to the logic.

[0094] The loss function used in the training is the cross-entropy loss function. The optimization goal is to maximize the conditional probability of the entire sequence of model output and the true labeled sequence. Through the driving of this loss function, the model parameters are constantly adjusted so that the probability distribution of each word unit in the generated response is as close as possible to the true sequence in the labeled data, thereby ensuring that the model not only outputs the correct structured result, but also generates a logical and consistent reasoning process with the labeled data. The joint action of these technical means ultimately enables the base multi-modal large model to learn the "positioning-abstracting-checking-judging" collaborative reasoning logic defined by the thinking chain information, and realizes the automation from perception to reasoning to structured output.

[0095] The fine-tuned model is used as the initial strategy, and a reinforcement learning algorithm based on grouping relative strategy optimization is used. Based on a set of multi-task reward functions, the steps of strategy optimization of the model include:

[0096] For each training sample, a plurality of candidate outputs are generated by sampling from the current model strategy;

[0097] Calculate the total reward of each candidate output under the multi-task reward function;

[0098] Based on the total reward of all candidate outputs, calculate the relative advantage within each group;

[0099] Based on the relative advantage as a guide, the model parameters are optimized through the policy gradient update algorithm, and KL divergence is used as a constraint term in the optimization process.

[0100] For each training sample, first, a plurality of candidate outputs are generated by sampling from the current model strategy to form a comparison group; then, the total reward score of each candidate output is calculated based on the multi-task reward function; then, by comparing the reward differences of each output in the group, the relative advantage is calculated as an optimization signal; finally, guided by the relative advantage, the model parameters are updated through the policy gradient algorithm. To prevent the policy update from being too large, KL divergence is introduced as a constraint term in the optimization process to ensure that the optimized strategy maintains appropriate continuity with the initial strategy, thereby improving the performance of the model while maintaining its reasoning stability.

[0101] The integrity state includes three states of clear, partially readable and unreadable;

[0102] For the chart area judged to be unreadable, output a preset unreadable identifier in the structured recognition result, and provide the reason for unreadability in the thinking chain reasoning chain.

[0103] The integrity state includes three states of clear, partially readable and unreadable, forming a hierarchical evaluation system of the readability of the chart area. When the model determines that a certain chart area is in an unreadable state, a preset unreadable identifier is output in the corresponding chart item of the structured recognition result, and the specific reason for causing unreadability is recorded in detail in the integrity judgment logic part of the thinking chain reasoning chain, thereby establishing a complete readability judgment and explanation mechanism.

[0104] In one possible embodiment, the present application takes the 2024 China new energy vehicle industry quarterly report as the processing object, and demonstrates the whole process from data preparation to model training and reasoning:

[0105] First, data construction, about 5000 pages of new energy vehicle industry research reports containing charts are selected as raw data, and all images are converted to PNG format with uniform resolution. Then, semi-automatic labeling process is used: first, the Layout-OCR model is used for layout analysis to locate the semantic structures such as tables and charts in the image and obtain their bounding box coordinates; then, the Qwen2.5-VL-7B model is used to deeply analyze the chart area positioned, and the title and content details of the chart are automatically generated; at the same time, the text area is recognized by OCR to generate the initial draft of the text. Finally, the professionals check and correct the above automatic output, and key supplement the complete thinking chain reasoning chain annotation, forming the final structured training sample.

[0106] Second, two-stage model training, the training process is divided into two core stages:

[0107] (1) Cold start supervised fine-tuning stage: using the data set constructed as described above, the LoRA parameter efficient fine-tuning method is used to fine-tune the Qwen2.5-VL-7B base model. The training hyperparameters are set as: batch size 8, training rounds 2, learning rate 2e-5, cross-entropy loss function is used, and training is terminated in advance when the validation set accuracy reaches 80%. During training, a specific prompt template is used to guide the model to learn the collaborative logic of "positioning-abstracting-OCR checking-integrity judgment". For example, for a certain sample, the model is trained to output the complete reasoning process and structured JSON result containing accurate bounding box, sales ranking abstract, OCR text and consistency check with chart data.

[0108] (2) Reinforcement learning optimization phase: take the model after cold start as the initial strategy, and use the grouping relative strategy optimization algorithm for further optimization. For each training image, 5 candidate outputs are generated by sampling. These outputs are evaluated by a comprehensive multi-task reward function, which quantifies scores from five dimensions of positioning accuracy, summary quality, OCR accuracy, reasoning consistency, and integrity judgment. For example, in one instance, the model obtained a high total reward of about 0.97 because the predicted bounding box had a high overlap with the true box, the summary was highly similar to the true content, the OCR error rate was low, the reasoning chain was completely consistent with the output, and the integrity judgment was correct. The algorithm then calculates the relative advantage of the candidate outputs within the group and uses it to guide the policy gradient update, while using KL divergence as a constraint to ensure training stability. After this stage of optimization, the model's comprehensive accuracy on multiple tasks improved to 80%.

[0109] Third, model reasoning and effect, input a new research report image that does not appear in the training set into the trained model for testing. As shown in the figure, the model successfully performed end-to-end structured recognition: for a pie chart that cannot be recognized due to scanning blur, the model not only accurately positioned its location, but also made the correct judgment of "unreadable information", and clearly stated the basis for the judgment in the reasoning chain; for the clear text area, it outputs high-fidelity OCR text. This output result fully proves the significant effect of the scheme in realizing accurate positioning, reliable rejection, and interpretable reasoning. Through quantitative evaluation, the model performs well in key indicators such as average accuracy of chart positioning, BLEU-4 score of summary, character error rate of OCR, and accuracy of unreadable rejection, meeting the high standards of precision and reliability required by financial research report automation analysis.

[0110] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0111] First, through the two-stage training method of cold start supervised fine-tuning + reinforcement learning optimization, the model learns to learn collaborative reasoning logic, effectively solving the problem of consistency between model reasoning and output, not only significantly improving the performance indicators of chart positioning, summary and other core tasks, but also breaking through the black box limitation of existing models that only give results without logic, and generating interpretable thought chains.

[0112] Second, a reward function that integrates positioning, summary, consistency and other multi-dimensional tasks is designed, which accurately describes the complex task requirements in research report structured understanding. This design solves the common problems of reward sparsity and bias in reinforcement learning training, can systematically optimize the comprehensive performance of the model on multiple tasks, and reduce manual intervention.

[0113] Third, a standardized data labeling style is defined, and a high-quality training dataset is constructed by incorporating thought chain information. This method can achieve model cold start with only thousands of samples, significantly reducing data costs and manual labeling burden compared to traditional methods, and has strong generalization ability to migrate to PDF, academic papers and other scenarios.

[0114] A possible implementation of a financial research report structured identification system based on a multi-modal large model is given below, which is used to perform each execution step and the corresponding technical effect of the financial research report structured identification method based on a multi-modal large model shown in the above embodiments and possible implementations, including:

[0115] A data construction module is configured to obtain multiple financial research report images and perform labeling to generate a training dataset;

[0116] A model training module is configured to perform two-stage training of the multi-modal large model based on the training dataset, including:

[0117] A cold start fine-tuning unit is configured to use thought chain information to supervise the fine-tuning of the base model;

[0118] A reinforcement learning optimization unit is configured to use the fine-tuned model as an initial strategy, use a grouping relative strategy optimization algorithm, and optimize the model based on a multi-task reward function;

[0119] A structured identification module is configured to input a financial research report image to be identified into the trained model and obtain a structured identification result including a chart list, a text, and a thought chain reasoning chain.

[0120] The above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal large model-based financial research report structuring identification method, characterized in that, include: Acquire multiple financial research report images and annotate each financial research report image to generate a training dataset. The annotation includes image labels, scene information, chart list, text, and thought chain reasoning chain. The chart list includes at least one chart item, and each chart item includes chart type, bounding box coordinates, title, content details, integrity status, and integrity reason. The steps involved in generating the thought chain and reasoning chain include: Using a layout analysis model, semantic structure localization is performed on images in financial research reports to obtain chart type and bounding box coordinates; The semantic structure located is parsed using a multimodal recognition model to generate the title and content details of the chart; Optical character recognition is performed on the text area of ​​a financial research report image to generate the text. By integrating chart types and bounding box coordinates, chart titles and content details, and body text, and supplementing them with manual verification, a thought chain and reasoning chain are generated. Based on the training dataset, a two-stage training process is performed on the multimodal large model; In the supervised fine-tuning phase of cold start, the base multimodal large model is supervisedly fine-tuned using the thought chain information in the labeled data, so that the base multimodal large model learns to follow the collaborative reasoning logic defined by the thought chain information and generates structured output; In the reinforcement learning optimization phase, the fine-tuned model is used as the initial policy. A reinforcement learning algorithm based on grouped relative policy optimization is adopted, and the model is optimized based on a set of multi-task reward functions. The multi-task reward function is a weighted sum of localization reward, summarization reward, OCR reward, consistency reward and integrity reward. The localization reward is calculated by gradient based on the intersection-union ratio (IUU) between the model-predicted bounding boxes and the ground truth bounding boxes. Summary rewards are calculated based on the BLEU scores of model-generated chart content details and actual content details. OCR rewards are calculated based on the character error rate between the text identified by the model and the actual text. Consistency rewards are used to verify the logical consistency between the thought chain and reasoning chain of the model output and the structured output content; The integrity reward is calculated based on the accuracy of the model's judgment of the unreadable state of the chart area; The image of the financial research report to be identified is input into the trained model to obtain the structured recognition result, which includes a list of charts, the main text, and the thought chain reasoning chain.

2. The financial research report structured recognition method of claim 1, wherein, Supervised fine-tuning of the base multimodal large model is performed using thought chain information from labeled data. This enables the base multimodal large model to learn to follow the collaborative reasoning logic defined by the thought chain information and generate structured outputs, including the following steps: A parameter-efficient fine-tuning method, LoRA, is used to fine-tune the base multimodal large model. LoRA adapters are added to the visual tower and language model head of the base multimodal large model. Training is performed using a prompt template that includes a thought chain and an inference chain. The prompt template includes a user input section and an assistant response section. The user input section includes an image and a task description. The task description requires the output of structured information in JSON format and a thought chain and inference chain. The assistant response section includes a thought chain and inference chain and JSON output. The loss function used in the training is a cross-entropy loss function to maximize the conditional probability of the output sequence given the true labels.

3. The financial research report structured recognition method of claim 1, wherein, The consistency reward includes a first sub-reward when the positioning logic matches the bounding box, a second sub-reward when the summary extraction depends on the content details, and a third sub-reward when the unreadable judgment reason matches the integrity reason. The integrity reward includes a highest reward when the model correctly labels the unreadable region with a reasonable reason, a zero reward when the model misjudges the readability, and a medium reward when the model correctly labels part of the readable region.

4. The financial research report structured recognition method of claim 1, wherein, Using the fine-tuned model as the initial strategy, a reinforcement learning algorithm based on grouped relative strategy optimization is used, and the steps of policy optimization of the model based on a set of multi-task reward functions include: For each training sample, a plurality of candidate outputs are generated from the current model policy; The total reward of each candidate output under the multi-task reward function is calculated; Based on the total reward of all candidate outputs, the relative advantage within each group is calculated; Using the relative advantage as a guide, the model parameters are optimized through a policy gradient update algorithm, and the KL divergence is used as a constraint term during optimization.

5. The financial research report structured recognition method of claim 1, wherein, The integrity state includes clear, partially readable, and unreadable; For the chart area judged as unreadable, a preset unreadable identifier is output in the structured recognition result, and the reason for unreadability is provided in the thought chain reasoning chain.

6. A multi-modal large model-based financial research report structured identification system, characterized in that, It includes: A data construction module is used to obtain multiple financial research report images and perform labeling to generate a training data set, including image identification, scene information, chart list, text, and thought chain reasoning chain. The chart list includes at least one chart item, each chart item containing chart type, bounding box coordinates, title, content details, integrity state, and integrity reason. The generation steps of the thought chain reasoning chain include: Using a layout analysis model to perform semantic structure positioning on the financial research report image to obtain chart type and bounding box coordinates; Using a multi-modal recognition model to analyze the positioned semantic structure to generate chart title and content details; Performing optical character recognition on the text area of the financial research report image to generate text; Integrating chart type and bounding box coordinates, chart title and content details, and text, and supplemented by manual verification, to generate a thought chain reasoning chain; A model training module is used to perform two-stage training on a multi-modal large model based on the training data set, which includes: A cold start fine-tuning unit is used to supervise the fine-tuning of the base model using thought chain information; A reinforcement learning optimization unit is used to use the fine-tuned model as the initial strategy, use a grouped relative strategy optimization algorithm, and perform policy optimization of the model based on a multi-task reward function, which is the weighted sum of positioning reward, summary reward, OCR reward, consistency reward, and integrity reward. The positioning reward is calculated based on the intersection over union of the model's predicted bounding box and the true bounding box. The summary reward is calculated based on the BLEU score of the model-generated chart content details and the true content details. The OCR reward is calculated based on the character error rate of the model-recognized text and the true text. The consistency reward is used to verify that the chain of thoughts of the model output is logically consistent with the structured output content; The integrity reward is based on the accuracy of the model's judgment of the unreadable state of the chart area; The structured identification module is configured to input the financial research report image to be identified into the trained model, and obtain a structured identification result including a chart list, a text of a body, and a chain of thoughts.

Images