A research report structuring method and system based on progressive layout fusion

Abstract

The application discloses a research report structuring method and system based on progressive layout fusion, and the method comprises the following steps: collecting research report data and processing the same to obtain a high-quality data set; performing layout analysis on the data of the high-quality data set through a layout analysis model, extracting layout features, compressing the layout features into feature vectors of a preset dimension, standardizing the feature vectors of the preset dimension, and obtaining pre-calculated layout feature vectors; designing a main loss function, an auxiliary loss function and a dynamic fusion weight function according to the pre-calculated layout feature vectors, and obtaining a total loss function; training a multi-modal large model to obtain a final fusion model; and inputting data images of research report data, set system prompt words and layout features into the fusion model to obtain structured data. The fusion model obtained through the double-loss dynamic optimization mechanism significantly improves the data information extraction accuracy and stability in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of research report structuring, and in particular to a research report structuring method and system based on progressive layout fusion. Background Technology

[0002] In the financial industry, massive amounts of unstructured or semi-structured documents (such as research reports, financial statements, insurance contracts, loan application forms, and business vouchers) constitute the core data source for information processing and business decision-making. Statistics show that 80% of financial institutions' business data exists in document form, with research reports serving as a crucial basis for investment decisions; their degree of structuring directly impacts investment research efficiency.

[0003] In recent years, visual-language models (VLMs) such as Qwen-VL, BLIP, and Flamingo have made significant progress in image understanding tasks. These models can process both visual and textual information simultaneously, theoretically making them suitable for document understanding tasks. These models can process text and image information end-to-end, avoiding the complex preprocessing of traditional pipelines and supporting flexible prompts and interactions, demonstrating strong multimodal understanding potential. However, general-purpose visual-language models still face significant limitations when directly applied to financial documents. They are insensitive to the specific layout of documents and struggle to accurately parse structured information such as tables and headings; their freely generated output methods also do not easily meet the strict format requirements for financial information extraction. Furthermore, these models often lack financial domain expertise and data-specific optimization, requiring substantial amounts of specialized training data and structural adjustments for practical application.

[0004] Page layout analysis technology aims to identify the physical structure and region types of documents, and its methods have evolved from rule-based and traditional machine learning to deep learning models. However, this technology has a fundamental limitation: it only focuses on the geometric division of page layout, which is severely disconnected from the semantic understanding of the content itself. This disconnect results in limited generalization ability, making it difficult to adapt to diverse and novel layouts, and it cannot directly complete end-to-end information extraction tasks, still requiring downstream modules for subsequent semantic parsing. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a research report structuring method based on progressive layout fusion.

[0006] Another objective of this invention is to provide a research report structuring system based on progressive layout fusion.

[0007] Another object of the present invention is to provide a server.

[0008] Another object of the present invention is to provide a computer-readable storage medium.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] A research report structuring method based on progressive layout fusion includes the following steps:

[0011] Collect and process research report data to obtain high-quality datasets;

[0012] The layout analysis model is used to perform layout analysis on high-quality datasets and extract layout features, which are then compressed into feature vectors of a preset dimension. The feature vectors of the preset dimension are then standardized to obtain pre-calculated layout feature vectors.

[0013] Based on the pre-calculated layout feature vectors, a main loss function, an auxiliary loss function, and a dynamic fusion weight function are designed to obtain the total loss function of the multimodal large model; the multimodal large model is then trained to obtain the final fusion model.

[0014] The data images of the research report, the set system prompts and layout features are input into the fusion model to obtain structured data.

[0015] Specifically, a research report structuring method based on progressive layout fusion includes the following steps:

[0016] S1. Collect research report data, preprocess the research report data to obtain preprocessed data, perform data annotation and three-level data augmentation on the preprocessed data to obtain a high-quality dataset; when annotating the preprocessed data, the system prompt words are set.

[0017] S2. Perform layout analysis on the high-quality dataset using the layout analysis model to obtain layout analysis results; extract layout features from the layout analysis results and compress them into feature vectors of preset dimensions; standardize the feature vectors of preset dimensions to obtain pre-calculated layout feature vectors.

[0018] S3. Based on the pre-calculated layout feature vectors, design the total loss function of the multimodal large model: When the multimodal large model iterates, calculate the visual language modeling loss function, the layout consistency loss function, and the dynamic fusion weight function, and obtain the total loss function based on the visual language modeling loss function, the layout consistency loss function, and the dynamic fusion weight function; the dynamic fusion weight function is adjusted according to different stages during the training of the multimodal large model; trainable low-rank adapters are inserted into the attention mechanism and feedforward network layer of the multimodal large model, and a fusion controller is formed through the dynamic fusion weight function; during the training of the multimodal large model, backpropagation is performed based on the total loss function, and only the parameters of the low-rank adapters and the fusion controller are updated, and the final fusion model is obtained when the training is completed;

[0019] S4. Input the data image of the research report data, the set system prompt words and layout features into the fusion model to obtain structured data.

[0020] In step S1, the preprocessing of the research report data involves converting the collected research report data into a high-quality RGB image format. The data formats include PDF, Word, and JPG. For PDF data, a preset resolution of at least 300 DPI is used for rendering. The converted data is then cleaned and deduplicated: blurry images, incomplete content, or irrelevant data are removed. Furthermore, cosine similarity calculation is used to remove highly repetitive content. The specific formula for calculating cosine similarity is as follows:

[0021] ;

[0022] in, and Given the feature vectors of two research report data images, when If the data exceeds the set threshold, the two research report data images are considered to be duplicates, and only one research report data is retained; finally, the preprocessed data is obtained.

[0023] In step S1, the annotation and enhancement of the preprocessed data involves: establishing a research report data field annotation system and setting system prompts; annotating the preprocessed data according to the research report data field annotation system and system prompts to obtain annotated data; the system prompts include role definition, field restrictions, value processing rules, output format, and prohibited items. The role definition clearly defines the multimodal large model as a professional financial document analysis assistant; field restrictions strictly limit the extraction of only specified fields, prohibiting the extraction of any other information; value processing rules return a string format for single-value fields, a list format for multi-value fields, and an empty string for non-existent fields; the output format is a strict JSON format containing all specified fields; and prohibited items prohibit modifying field names, concatenating multiple values, summarizing or rewriting the original research report data content.

[0024] A three-level document-aware enhancement strategy was applied to the labeled data for data augmentation. This strategy included geometric transformation, color adjustment, and noise addition. Geometric transformation was a mild enhancement, simulating scanning tilt, shooting angle, distance changes, and paper deformation through controlled random rotation, perspective, scaling, and cropping operations. Color adjustment was a medium enhancement, restoring the visual effects under different lighting and print quality conditions by adjusting brightness and contrast and applying color dithering. Noise addition was a severe enhancement, introducing Gaussian, salt-and-pepper, and speckle noise to reproduce electronic noise scenarios such as sensor interference and transmission errors. All newly generated blank areas were filled with a white background during the document-aware enhancement of the labeled data, resulting in the enhanced data, i.e., a high-quality dataset.

[0025] Step S2 involves performing layout analysis on each research report data image in the high-quality dataset using a layout analysis model: detecting content regions in the research report data images and outputting a set of bounding boxes. for:

[0026] ;

[0027] ;

[0028] in, These are the x and y coordinates of the top-left corner of the content area, respectively. These are the width and height of the content area, respectively. For content area type, To detect the confidence score, The spatial location of the content area;

[0029] Structural relationship analysis is performed based on the spatial location and type of content areas to infer the logical relationships between content areas. These logical relationships include inclusion, adjacency, and hierarchy. Specifically, inclusion is when a title includes the body paragraph below it; adjacency is when a table is paired with its corresponding figure caption; and hierarchy is when chapter titles are nested. The final layout analysis results are obtained.

[0030] Layout features are extracted from the layout analysis results, and the feature vectors of these features are compressed into feature vectors of a preset dimension. These preset dimension feature vectors include type distribution features, spatial distribution features, structural complexity features, page attribute features, and relationship features. Among these, the type distribution feature calculates the proportion of each type within the content area in the research report data.

[0031] ;

[0032] in, For content area type, , For indicator functions, Number of content area types;

[0033] The feature vectors of a preset dimension are normalized to the [0,1] interval. The specific calculation formula is as follows:

[0034] ;

[0035] in, For the feature components in a feature vector of a preset dimension, and The minimum and maximum values ​​of this feature component are obtained from the high-quality dataset; finally, the pre-computed layout feature vector is obtained.

[0036] In step S3, the total loss function for designing the multimodal large model is: calculating the main loss function, which is the visual language modeling loss function, and the specific calculation formula is:

[0037] ;

[0038] in, For multimodal large model parameters; The input is obtained by combining research report data images with labeled text. The target sequence is a JSON-formatted string output during data labeling. The length of the target sequence. To predict the probability of the next label for a multimodal large model given historical labels and input;

[0039] Calculate the auxiliary loss function, which is the layout consistency loss function, and the specific calculation formula is as follows:

[0040] ;

[0041] in, For the pre-computed layout feature vector, These are layout-related features extracted from the intermediate layers of a multimodal large model. For batch size, for Norm;

[0042] The dynamic fusion weight function is calculated using the following formula:

[0043] ;

[0044] in, This is the current Epoch number. This represents the total number of Epochs, where Epoch is the number of iterations of the multimodal large model; among them, 0.00005, 0.0002, and 0.0001 are weight values, which can be adjusted according to different stages of training the multimodal large model;

[0045] When the multimodal large model is not iterated (i.e., Epoch=0), the total loss function equals the main loss function; when the multimodal large model is iterated (i.e., Epoch≥1), the total loss function is:

[0046] ;

[0047] The parameters are updated based on the main loss function and the auxiliary loss function. The specific calculation formula is as follows:

[0048] ;

[0049] in, For learning rate, These are the parameters of the low-rank adapter and fusion controller at the t-th iteration. For parameters The gradient operator.

[0050] Another objective of this invention is achieved through the following technical solution:

[0051] A research report structuring system based on progressive layout fusion includes a client and a server. The server includes a document preprocessing module, a model inference service module, and an instruction generation and result encapsulation module.

[0052] The document preprocessing module receives research report data uploaded by users in the client, sets system prompt words based on the research report data, preprocesses the research report data, and obtains preprocessed data.

[0053] Model inference service module: Obtain layout features from preprocessed data, input the preprocessed data, layout features and system prompt words into the fusion model, and the fusion model outputs a structured string in JSON format. The fusion model is a trained multimodal large model.

[0054] Instruction generation and result encapsulation module: After validating the structured string in JSON format output by the fusion model, it encapsulates it into a standardized API response and returns it to the client;

[0055] Once the client receives the standardized API response, it can directly use the structured data without additional parsing.

[0056] Another objective of this invention is achieved through the following technical solution:

[0057] A server includes a processor and a memory, the memory storing at least one program that is loaded and executed by the processor to implement the above-described research report structuring method based on progressive layout fusion.

[0058] Another objective of this invention is achieved through the following technical solution:

[0059] A computer-readable storage medium storing at least one program, which is loaded and executed by a processor to implement the above-described research report structuring method based on progressive layout fusion.

[0060] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0061] 1. This invention integrates visual semantics and layout structure. The integrated model can not only understand text content, but also make logical inferences based on the layout, which significantly improves the accuracy and stability of information extraction in scenarios such as complex layouts, multiple tables, and mixed text and images.

[0062] 2. This invention uses an offline method to extract and compress the layout features of research report data into feature vectors of preset dimensions, covering multiple dimensions of information such as type distribution, spatial structure, and page attributes. This not only preserves key structural information but also greatly improves training efficiency and feature consistency.

[0063] 3. This invention designs a three-level intensity data augmentation method based on the characteristics of financial research reports, and combines deduplication and cleaning mechanisms to construct a high-quality and diverse training dataset, thereby enhancing the model's robustness to layout variations and quality degradation in real-world scenarios.

[0064] 4. The dual-loss dynamic optimization mechanism of this invention combines visual language modeling loss and layout consistency loss, and achieves multi-objective collaborative optimization through dynamic weight function, so that the model can maintain sensitivity and consistency of document layout while understanding the semantics of the content.

[0065] 5. This invention achieves end-to-end structured output, with the model directly outputting standardized JSON data without relying on subsequent rule parsing or entity linking modules, simplifying the system process and improving processing efficiency and system reliability. Attached Figure Description

[0066] Figure 1 The graph shows how the training loss of the fusion model changes with the number of steps.

[0067] Figure 2 To smooth the training loss map for the fusion model.

[0068] Figure 3 This is a graph showing how the training loss of the baseline model changes with the number of steps.

[0069] Figure 4 This is a smoothed training loss graph for the baseline model.

[0070] Figure 5 This is an interaction diagram of a research report structured system based on progressive layout integration.

[0071] Figure 6 Flowchart for pre-computing layout features.

[0072] Figure 7 This is a flowchart of the training process for a multimodal large model. Detailed Implementation

[0073] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0074] A research report structuring method based on progressive layout fusion includes the following steps:

[0075] S1. Collect research report data, preprocess the research report data to obtain preprocessed data, perform data annotation and three-level data augmentation on the preprocessed data to obtain a high-quality dataset; when annotating the preprocessed data, the system prompt words are set.

[0076] S2. Perform layout analysis on the high-quality dataset using the layout analysis model to obtain layout analysis results; extract layout features from the layout analysis results and compress them into 32-dimensional feature vectors; standardize the 32-dimensional feature vectors to obtain pre-calculated layout feature vectors.

[0077] S3. Based on the pre-calculated layout feature vectors, design the total loss function of the multimodal large model: When the multimodal large model iterates, calculate the visual language modeling loss function, the layout consistency loss function, and the dynamic fusion weight function, and obtain the total loss function based on the visual language modeling loss function, the layout consistency loss function, and the dynamic fusion weight function; the dynamic fusion weight function is adjusted according to different stages during the training of the multimodal large model; trainable low-rank adapters are inserted into the attention mechanism and feedforward network layer of the multimodal large model, and a fusion controller is formed through the dynamic fusion weight function; during the training of the multimodal large model, backpropagation is performed based on the total loss function, and only the parameters of the low-rank adapters and the fusion controller are updated, and the final fusion model is obtained when the training is completed;

[0078] S4. Input the data image of the research report data, the set system prompt words and layout features into the fusion model to obtain structured data.

[0079] To construct a high-quality dataset, we extensively collected diverse research reports from public and authorized channels. Regarding data standards, we focused on reports from the past five years to ensure data coverage of different market cycles and major industries such as finance, technology, consumer goods, pharmaceuticals, and manufacturing. The collected document types were comprehensive, including in-depth reports, industry research, company surveys, and strategy reports. All documents were required to be in PDF format with complete text, tables, and charts to ensure the richness and authenticity of the information, providing a solid data foundation for model training.

[0080] The collected research report data is converted into high-quality RGB image format, including PDF, Word, and JPG formats. For PDF data, a preset resolution of at least 300 DPI is used for rendering. The converted data is then cleaned and deduplicated: blurry images, incomplete content, or irrelevant data are removed. Cosine similarity is then used to remove highly repetitive data. The specific formula for calculating cosine similarity is as follows:

[0081] ;

[0082] in, and Given the feature vectors of two research report data images, set a threshold. ,when At that time, it was assumed that the two research report data images were duplicates, and only one of the research report data was retained; finally, the preprocessed data was obtained.

[0083] Establish a complete research report field annotation system, including: basic research report information, stock information, investment recommendations, financial data, and text content. The basic research report information includes the title, publication date, issuing institution, and analyst; stock information includes the stock code, stock name, and industry; investment recommendations include the investment rating (buy / overweight / hold / underweight / sell), target price, and rating change; financial data includes operating revenue, net profit, earnings per share, and price-to-earnings ratio; and text content includes core viewpoints, investment logic, and risk warnings. The annotation format is the structured text expected from the multimodal large model output, in JSON format. For example: {"Stock Code": "YYY", "Investment Rating": "Buy"}.

[0084] The system prompts specifically designed for structured research report tasks include role definitions, field restrictions, value processing rules, output format, and prohibited actions. The role definition clearly defines the multimodal large model as a professional financial document analysis assistant. Field restrictions strictly limit extraction to only specified fields, prohibiting the extraction of any other information. Value processing rules return single-valued fields as strings, multi-valued fields as lists, and non-existent fields as empty strings. The output format is a strict JSON format, containing all specified fields. Prohibited actions include prohibiting modification of field names, concatenation of multiple values, and summarizing or rewriting the original research report data. Finally, the preprocessed data is labeled according to the research report data field labeling system and the system prompts to obtain labeled data.

[0085] A three-level (mild, moderate, and severe) document-aware augmentation strategy was employed to augment the labeled data, thereby improving the robustness and generalization ability of the multimodal large model. This strategy simulates various physical deformations and quality degradations that documents may encounter in real-world data acquisition and processing through systematic geometric transformations, color adjustments, and noise addition.

[0086] Specifically, geometric transformations effectively simulate scanning tilt, shooting angle, distance changes, and paper deformation through controllable random rotation, perspective, scaling, and shearing operations; color transformations restore the visual effects under different lighting and print quality conditions by adjusting brightness and contrast and performing color dithering; and noise addition reproduces electronic noise scenarios such as sensor interference and transmission errors by introducing Gaussian, salt-and-pepper, and speckle noise. All operations fill newly generated blank areas with a white background to generate diverse and highly realistic training samples while strictly maintaining the core characteristics of the document itself.

[0087] For the already labeled data, the above three-level augmentation operations are performed through automated scripts, thereby expanding the scale and increasing the diversity of the training dataset. This data augmentation process systematically generates various augmented samples simulating real-world scenarios without relying on additional manual annotation costs, effectively alleviating the overfitting problem caused by insufficient or monotonous training data. The augmented dataset further improves the model's adaptability and recognition accuracy under different noise environments, visual deformations, and lighting changes, providing sufficient and high-quality data support for building a robust document analysis and recognition system, ultimately resulting in a high-quality dataset.

[0088] Regarding the selection of a multimodal large-scale model, this invention adopts the Qwen3-VL-2B multimodal large-scale language model as the base model. This model employs an efficient multimodal architecture, mainly composed of a visual encoder, a projection layer, and a large-scale language model core. The visual encoder uses a Vision Transformer (ViT) structure and is responsible for efficiently extracting deep semantic features from the input image. The projection layer is typically a multilayer perceptron (MLP) and is responsible for mapping visual features to a semantic space aligned with text features. The large-scale language model core (LLM) serves as the reasoning center for text understanding and generation, receiving and fusing aligned multimodal information for final decision-making.

[0089] The core consideration in choosing this model lies in its comprehensive advantages in resource efficiency, training stability, and task adaptability:

[0090] High resource efficiency: Its moderate scale of 2 billion parameters significantly reduces the demand for computing resources (such as GPU memory) and energy consumption during the training and inference stages compared to ultra-large-scale models, and shortens the model fine-tuning and iteration cycle.

[0091] Stable training process: The smaller number of parameters reduces model complexity, resulting in faster convergence when fine-tuning for specific task data, effectively mitigating the risk of overfitting, and improving the reliability and success rate of the training process.

[0092] High adaptability to vertical tasks: Although it may not be as general as models with a larger number of parameters, through efficient fine-tuning, Qwen3-VL-2B can quickly adapt to data in specific domains, achieving a balance between high accuracy and high inference efficiency on target vertical tasks. This makes it easier to deploy and integrate in real-world application scenarios, meeting the comprehensive design requirements of this invention for technical practicality, economy, and efficiency.

[0093] In terms of layout analysis, this invention employs the PP-DocLayout_plus-L model for feature extraction. This model is an advanced tool specifically optimized for document analysis, with its outstanding advantage lying in its precise understanding of document-specific structures. It can accurately identify 11 document regions, including body text, headings, tables, and charts, and demonstrates excellent recognition capabilities for complex tables and unique elements (such as investment rating boxes) commonly found in financial research reports. Furthermore, it achieves a good balance between recognition accuracy, processing speed, and resource consumption, providing not only region locations but also resolving the hierarchical relationships between regions, thus providing a solid and efficient structured foundation for subsequent semantic understanding.

[0094] like Figure 6As shown, layout feature estimation involves performing layout structure analysis on the document image before model training, and converting and caching the resulting complex visual layout information (such as the type, position, and hierarchical relationship of each region) into feature vectors. It employs an offline computation, one-time analysis, and multiple reuse strategy. The specific process is as follows:

[0095] The PP-DocLayout_plus-L model is used to perform layout analysis on each research report image in a high-quality dataset: detecting content regions in the research report images and outputting a set of bounding boxes. for:

[0096] ;

[0097] ;

[0098] in, These are the x and y coordinates of the top-left corner of the content area, respectively. These are the width and height of the content area, respectively. For content area type, To detect the confidence score, The spatial location of the content area;

[0099] Structural relationship analysis is performed based on the spatial location and type of content areas to infer the logical relationships between content areas. These logical relationships include inclusion, adjacency, and hierarchy. Inclusion is when a title includes the body paragraph below it; adjacency is when a table is paired with its corresponding figure caption; and hierarchy is when chapter titles are nested. The final layout analysis results are obtained.

[0100] Layout features are extracted from the layout analysis results, and the feature vectors of the layout features are compressed into 32-dimensional feature vectors. The 32-dimensional feature vectors include type distribution features (6-dimensional), spatial distribution features (8-dimensional), structural complexity features (8-dimensional), page attribute features (6-dimensional), and relationship features (4-dimensional).

[0101] Among them, the distribution of the proportion of each type in the research report data is calculated in the content area of ​​the type distribution feature:

[0102] ;

[0103] in, For content area type, , For indicator functions, Number of content area types;

[0104] Spatial distribution characteristics include the horizontal and vertical distribution of the region center point (mean and standard deviation), statistical characteristics of region size (average aspect ratio, coefficient of variation), and page coverage (the proportion of the total area of ​​all regions to the total page area); structural complexity characteristics include the number of regions. The normalized value, region density (number of regions per unit area), arrangement order (an assessment based on region alignment), and layout entropy (a measure of the regularity of the layout) are all considered. Page attribute features include page aspect ratio, percentage of effective content area, statistics of blank edge areas, and symmetry index. Relationship features include average nesting depth, average distance between regions, type co-occurrence pattern, and structural consistency score.

[0105] The 32-dimensional feature vector is normalized to the [0,1] interval. The specific calculation formula is as follows:

[0106] ;

[0107] in, For the feature components in a feature vector of a specified dimension, and The minimum and maximum values ​​of this feature component are obtained from the high-quality dataset; finally, the pre-computed layout feature vector is obtained.

[0108] The pre-computed layout feature vectors are cached and stored using key-value pairs, where the key is the hash value of the image content and the value is a 32-dimensional feature vector. It supports fast lookup and batch loading, and provides a feature quality verification mechanism to automatically detect and repair abnormal features.

[0109] Layout feature pre-computation is separated from the training loop and completed in advance, drastically reducing feature loading time from hundreds of milliseconds to milliseconds, thereby improving overall training efficiency. Simultaneously, it ensures that all training and inference samples extract features through a unified and stable process, avoiding data inconsistencies caused by differences in layout analysis model versions or states, and guaranteeing the stability of the feature space.

[0110] In terms of model training implementation, to efficiently implement the progressive layout fusion strategy and reduce training costs, this invention, after loading the pre-trained Qwen3-VL-2B pedestal model weights, inserts trainable low-rank adapters into its attention mechanisms (such as Q, K, and V projection layers) and key linear layers of the feedforward network. Within this framework, the training process for progressive layout fusion only updates the parameters of these low-rank adapters and the controller parameters specifically designed for fusion layout features, while freezing most of the weights of the original pedestal model. This reduces the number of trainable parameters to 0.1%–1% of the original model, significantly improving training efficiency and reducing memory consumption, while effectively avoiding overfitting that may be caused by full parameter fine-tuning, ensuring the stability of the training process, and enabling the model to focus more on learning the progressive fusion rules between "semantic understanding" and "layout awareness."

[0111] like Figure 7 As shown, when training a large multimodal model, traditional multimodal fusion methods typically employ fixed weights or simple decay strategies for the loss function. After introducing layout features, directly using the original visual-language model's loss function can easily lead to training instability or feature interference. To avoid this problem, a progressive fusion mechanism is introduced, dynamically adjusting the fusion weights of layout features at different stages of training.

[0112] The main loss function is calculated, which is the visual language modeling loss function. The specific calculation formula is as follows:

[0113] ;

[0114] in, For multimodal large model parameters; The input is obtained by combining the research report data images with the text of the labeled data; The target sequence is a JSON-formatted string output during data labeling. The length of the target sequence. To predict the probability of the next label for a multimodal large model given historical labels and input;

[0115] Calculate the auxiliary loss function, which is the layout consistency loss function, and the specific calculation formula is as follows:

[0116] ;

[0117] in, For the pre-computed layout feature vector, These are layout-related features extracted from the intermediate layers of a multimodal large model. For batch size, for Norm;

[0118] The dynamic fusion weight function is calculated using the following formula:

[0119] ;

[0120] in, This is the current Epoch number. This represents the total number of Epochs, where Epoch is the number of model iterations. Note that the weight values ​​(0.00005, 0.0001, etc.) are adjusted based on the current training data and can be adjusted according to the model size, dataset, and task requirements in practical applications.

[0121] When the multimodal large model is not iterated (i.e., Epoch=0), the total loss function equals the main loss function; when the multimodal large model is iterated (i.e., Epoch≥1), the total loss function is:

[0122] ;

[0123] This loss function goes through four stages:

[0124] Initial stage: Do not use layout features at all, allowing the model to focus on learning the basic abilities of visual language tasks;

[0125] Introduction phase: Layout features are introduced with extremely small weights (0.00005-0.0001) as auxiliary signals;

[0126] Enhancement phase: Gradually increase the fusion weight (0.0001-0.0002) to strengthen layout awareness;

[0127] Stable phase: Maintain a relatively stable fusion weight (0.0002-0.0003) to balance the two features.

[0128] The parameters are updated based on the main loss function and the auxiliary loss function. The specific calculation formula is as follows:

[0129] ;

[0130] in, For learning rate, Let be the parameters of the low-rank adapter and the fusion controller at the t-th iteration. For parameters The gradient operator.

[0131] The fusion controller is composed of the calculation results of the dynamic fusion weight function, which is a dynamic weight calculation unit. It calculates the current layout feature fusion weight based on the current training cycle number and the dynamic fusion weight function.

[0132] Backpropagation is performed based on the total loss function, updating only the low-rank adapter parameters and the trainable parameters in the fusion controller, while most parameters of the base model remain frozen, achieving end-to-end optimization. Iterative training is then performed on the large multimodal model, repeating the above process until the preset number of training epochs or convergence criteria are reached. Finally, the trained model, i.e., the fusion model, is saved.

[0133] To verify the effectiveness of the progressive layout fusion strategy during training, we compared the changes in training loss between using only the original Qwen3-VL-2B model (hereinafter referred to as the "baseline model") and the fused model under the same training conditions.

[0134] Depend on Figures 1 to 4 It can be seen that the training loss curve of the baseline model shows relatively obvious fluctuations in the middle of training (about 50-150 steps), which reflects the instability of the model in learning complex visual-language alignment tasks; the training loss curve of the fusion model is smoother overall, and the fluctuation amplitude is significantly reduced, indicating that the progressive layout fusion strategy effectively stabilizes the training process and makes the model learning more robust.

[0135] The baseline model exhibits several local fluctuations in its loss descent process, which may be due to the model's difficulty in adapting to complex layouts when relying solely on semantic information. The fusion model, by introducing layout consistency constraints, provides the model with additional structured supervision signals, enabling the model to perceive and adapt to the document's layout structure while understanding the document content, thus achieving a smoother convergence trajectory.

[0136] The progressive layout fusion strategy introduces layout features with minimal weight in the early stages of training, avoiding feature interference that may occur in the early training. As training progresses, the fusion weight of layout features gradually increases, and the fusion model gradually learns the correlation between page structure and semantic content. This phased fusion method not only preserves the model's original semantic understanding ability but also enhances its perception and adaptation to document layout.

[0137] Figures 1 to 4 The final convergence levels were similar (both decreased to about 0.2), indicating that both methods can effectively learn structured tasks. However, the fusion model showed better training stability while maintaining similar final performance, which means more reliable model performance and less need for hyperparameter tuning in practical applications.

[0138] The comparative analysis of the training results clearly demonstrates that the progressive layout fusion strategy proposed in this invention, by introducing layout supervision signals in stages and with dynamic balance, effectively solves the training instability problem caused by the lack of layout information in pure semantic models during document understanding tasks. The smooth convergence of the training process not only means that the fusion model is more likely to achieve good performance, but also reflects more profoundly that the fusion model has learned more robust feature representations that simultaneously contain semantic and layout information. This enables the fusion model to truly collaboratively understand the "content" and "form" of a document, thus exhibiting stronger generalization ability and higher information extraction accuracy when facing financial documents with diverse layouts and complex structures in real-world scenarios.

[0139] This empirical result technically verifies the effectiveness and superiority of the progressive layout fusion methodology, providing a key guarantee for achieving high-precision and robust end-to-end document structuring, and laying a solid foundation for the stable deployment and high-performance of the fusion model in real business scenarios.

[0140] like Figure 5 As shown, a research report structuring system based on progressive layout fusion includes a client and a server. The server includes a document preprocessing module, a model inference service module, and an instruction generation and result encapsulation module.

[0141] The document preprocessing module receives research report data uploaded by users in the client, sets system prompt words based on the research report data, preprocesses the research report data, and obtains preprocessed data.

[0142] Model inference service module: Obtain layout features from preprocessed data, input the preprocessed data, layout features and system prompt words into the fusion model, and the fusion model outputs a structured string in JSON format. The fusion model is a trained multimodal large model.

[0143] Instruction generation and result encapsulation module: After validating the structured string in JSON format output by the fusion model, it encapsulates it into a standardized API response and returns it to the client;

[0144] Once the client receives the standardized API response, it can directly use the structured data without additional parsing.

[0145] The specific process is as follows:

[0146] Client request submission: The user uploads the document to be processed (such as a research report in PDF format) and can choose to provide a custom list of extracted fields (schema). If not provided, the system uses a predefined default set of fields. The system automatically constructs clear and structured prompts and instructions based on the schema provided by the user.

[0147] Example: If the schema is ["Title", "Company Name", "Stock Code", "Investment Rating"], the following command will be generated:

[0148] You are a professional financial document analysis assistant, strictly adhering to the following rules to extract information:

[0149] [Core Commands]

[0150] 1. Strict restrictions: Only extract these specified fields: ["Title", "Company Name", "Stock Code", "Investment Rating"], absolutely no other fields will be extracted.

[0151] 2. Field does not exist: If the value of a field does not exist, strictly return an empty string "".

[0152] 3. Value format:

[0153] - Single value → String format

[0154] - Multiple values ​​→ List format

[0155] 4. Value processing: Keep the original text intact, without any modification, summarization, or splicing.

[0156] [Strictly Prohibited]

[0157] - Do not add any extra fields.

[0158] - Do not modify the key name

[0159] - Do not use plus signs, commas, etc. to concatenate multiple values.

[0160] - It is forbidden to summarize, interpret, or rewrite the value.

[0161] Output Format

[0162] The output must strictly adhere to the following JSON format and contain only the specified fields:

[0163] {

[0164] "key1": "value", # Single value: string

[0165] "key2": ["value1", "value2"], # Multiple values: list

[0166] "key3": "" # Does not exist: empty string

[0167] }

[0168]

Example

[0169] Input fields: ["Figure A_Title", "Figure B_Column Name_1QFY24"]

[0170] Output: {"Figure A_Title": "Revenue and YoY / QoQ Growth Rate for XXX Quarter", "Figure B_Column Name_1QFY24": ["7,192", "-13.2%", "18.9%", "4,648", "64.6%", "2,043", "26.3%", "28.4%"]}

[0171] Now we will begin processing, strictly adhering to the above rules, and only outputting the required JSON format.

[0172] The preprocessed data and generated instructions are concatenated into a multimodal input, which is then fed into the fusion model for end-to-end inference. The fusion model, through a vision-language fusion mechanism, simultaneously understands the image content and instruction semantics, locates and extracts target fields, and outputs a structured string conforming to JSON format. After validation, this string is encapsulated into a standardized API response and returned to the client. Upon receiving the response, the client can directly use the structured data without additional parsing. An example of the response format is shown below:

[0173] {

[0174] "status": "success",

[0175] "data": {

[0176] Title: Strong Downstream Demand for Power and Energy Storage Drives High Shipments of Lithium Iron Phosphate

[0177] Company Name: XXX

[0178] Stock Code: "YYY"

[0179] Investment Rating: Buy (Maintain)

[0180] }

[0181] }

[0182] The system of this invention is flexible and configurable, supports user-defined extraction fields, and adapts to the needs of different business scenarios; it can achieve high concurrency support, and based on inference optimization frameworks such as VLLM, it can achieve batch processing and low-latency response; it can automate the entire process from document upload to structured output without manual intervention; it is easy to integrate, and provides a standardized RESTful API, which can be easily connected to existing investment research systems, knowledge bases or business process platforms.

[0183] Meanwhile, this invention provides:

[0184] A server includes a processor and a memory, the memory storing at least one program that is loaded and executed by the processor to implement the above-described research report structuring method based on progressive layout fusion.

[0185] A computer-readable storage medium storing at least one program that is loaded and executed by a processor to implement the above-described research report structuring method based on progressive layout fusion.

[0186] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A research report structuring method based on progressive layout fusion, characterized in that, Includes the following steps: Collect and process research report data to obtain high-quality datasets; The layout analysis model is used to perform layout analysis on high-quality datasets and extract layout features, which are then compressed into feature vectors of a preset dimension. The feature vectors of the preset dimension are then standardized to obtain pre-calculated layout feature vectors. Based on the pre-calculated layout feature vectors, a main loss function, an auxiliary loss function, and a dynamic fusion weight function are designed to obtain the total loss function of the multimodal large model; the multimodal large model is then trained to obtain the final fusion model. The data images of the research report, the set system prompts and layout features are input into the fusion model to obtain structured data; The process of obtaining the total loss function of the multimodal large model specifically includes: calculating the main loss function, which is the visual language modeling loss function, and the specific calculation formula is as follows: ； in, For multimodal large model parameters; The input is obtained by combining the research report data images with the text of the labeled data; The target sequence is a JSON-formatted string output during data labeling. The length of the target sequence. To predict the probability of the next label for a multimodal large model given historical labels and input; Calculate the auxiliary loss function, which is the layout consistency loss function, and the specific calculation formula is as follows: ； in, For the pre-computed layout feature vector, These are layout-related features extracted from the intermediate layers of a multimodal large model. For batch size, for Norm; The dynamic fusion weight function is calculated using the following formula: ； in, This is the current Epoch number. This represents the total number of Epochs, where Epoch is the number of iterations in a multimodal large model. When the multimodal large model is not iterated (i.e., Epoch=0), the total loss function equals the main loss function; when the multimodal large model is iterated (i.e., Epoch≥1), the total loss function is: ； The parameters are updated based on the main loss function and the auxiliary loss function. The specific calculation formula is as follows: ； in, For learning rate, Let be the parameters of the multimodal large model at the t-th iteration. For parameters The gradient operator.

2. The research report structuring method based on progressive layout fusion according to claim 1, characterized in that, The pre-computed layout feature vector is obtained in the following way: Detect content regions in research report data images; Structural relationship analysis is performed based on the spatial location and type of content areas to infer the logical relationships between content areas, and finally the layout analysis results are obtained. Layout features are extracted from the layout analysis results, and the feature vectors of the layout features are compressed into feature vectors of preset dimensions. The feature vectors of preset dimensions include type distribution features, spatial distribution features, structural complexity features, page attribute features, and relationship features. Normalize the feature vectors of the preset dimensions to the [0,1] interval.

3. A research report structuring system based on progressive layout fusion, characterized in that, Used to implement the report structuring method based on progressive layout fusion as described in any one of claims 1 to 2; It includes a client and a server, wherein the server includes a document preprocessing module, a model inference service module, and an instruction generation and result encapsulation module; The document preprocessing module receives research report data uploaded by users in the client, sets system prompt words based on the research report data, preprocesses the research report data, and obtains preprocessed data. Model inference service module: Obtain layout features from preprocessed data, input the preprocessed data, layout features and system prompt words into the fusion model, and the fusion model outputs a structured string in JSON format. The fusion model is a trained multimodal large model. Instruction generation and result encapsulation module: After validating the structured string in JSON format output by the fusion model, it encapsulates it into a standardized API response and returns it to the client; Once the client receives the standardized API response, it can directly use the structured data without additional parsing.

4. A server, characterized in that, The server includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the report structuring method based on progressive layout fusion as described in any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is loaded and executed by a processor to implement the report structuring method based on progressive layout fusion as described in any one of claims 1 to 2.

Images