Multi-modal analysis method for mechanism investigation PDF (Portable Document Format)

Abstract

The invention discloses a mechanism investigation PDF-oriented multi-modal analysis method, and relates to the technical field of electrical digital data processing. The method comprises the following steps: step 1, analyzing a PDF file based on a layout segmentation model to obtain a structured region and an unstructured region; 2, processing the table of the structured area through a cross-page table recombination algorithm to obtain complete table data, extracting through a rule template matching algorithm to obtain key fields, and forming structured data; 3, the format self-attention enhancement model comprises an input layer, a Transform coding layer and a pre-training task module which are connected in sequence, a cross-modal attention mechanism and a multi-modal mask pre-training task are adopted in the pre-training stage, and layered fine adjustment and adversarial training are adopted in the fine adjustment stage; 4, the unstructured area is processed by a format self-attention enhancement model to obtain modularized information; through a format self-attention enhancement model and the like, high analysis efficiency, good integrity and good accuracy are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a multimodal analysis method for PDFs of institutional surveys. Background Art

[0002] Currently, during the research process, organizations publish PDF documents containing information such as the research method, timeframe, organization, reception staff, and content. These documents are typically presented in a question-and-answer format. Existing technology typically relies on traditional optical character recognition (OCR) to parse PDF files, extracting and recording key information through manual reading and review.

[0003] The application, published as CN118747902A, is titled "OCR-Based Table Text Recognition Method and System," and includes the following steps: parsing a PDF file and converting it into image data; and performing layout analysis on the image data to obtain layout analysis results. Traditional parsing methods, such as tools like PyPDF2 and optical character recognition (OCR) technology, parse PDF files and extract text objects directly from the content stream. However, due to their reliance on font encoding mapping to resolve character encoding issues, these applications often face multiple limitations, resulting in incomplete parsing of complex elements such as tables and images, difficulty in connecting content across pages, and other issues. Parsing accuracy is low, with 90% relying on manual review.

[0004] The application, published as CN119360398A, is titled "Efficient Semantic Understanding of PDF Content Using Deep Learning." The method involves parsing multimodal PDF data from the construction industry, learning modal structured information, and constructing a segmentation and extraction module. The module analyzes the PDF modal distribution and extracts structured information. However, the PDF parsing process is slow, with invalid information taking up time. The method also relies heavily on manual review, resulting in low efficiency.

[0005] Analyzing the above two documents and existing technical solutions, the existing technical solutions have the following technical problems.

[0006] Limitations of traditional optical character recognition (OCR) technology: It relies on a single rule or general model and cannot distinguish between structured and unstructured areas, resulting in information fragmentation. Structured areas include tables and headers, and unstructured areas include free text.

[0007] Cross-page table processing issues: Existing solutions, such as coordinate matching, cannot solve the problem of semantic fragmentation, resulting in the loss of key data.

[0008] Poor domain adaptability: General deep learning natural language processing models have low recognition rates for financial terms such as operating income and inventory turnover.

[0009] Inefficiency: Manual review requires checking each field one by one, which is time-consuming and prone to errors. Summary of the Invention

[0010] The present invention provides a multimodal parsing method for PDF files of institutional research, which solves the technical problems of low parsing efficiency, poor integrity and poor accuracy of PDF files.

[0011] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0012] A multimodal analysis method for institutional research PDFs, including the following steps:

[0013] Step 1: Parse the PDF file based on the layout segmentation model to obtain structured and unstructured regions. Structured regions include tables, headers, and footers, while unstructured regions include free text.

[0014] Step 2: The table in the structured area is processed by the cross-page table reorganization algorithm to obtain complete table data. The table data is extracted by the rule template matching algorithm to obtain key fields and form structured data;

[0015] Step 3: The layout self-attention enhancement model includes a sequentially connected input layer, a Transformer encoding layer, and a pre-training task module. In the pre-training stage, a cross-modal attention mechanism and a multimodal mask pre-training task are used. In the fine-tuning stage, layer-by-layer fine-tuning and adversarial training are used to obtain the layout self-attention enhancement model.

[0016] Step 4: The unstructured area is processed by the layout self-attention enhancement model to obtain modular information.

[0017] A further technical solution is that in step 2, the cross-page table reorganization algorithm processing steps include breakpoint detection, calculating the cell similarity score based on the format and sentence features of the PDF file, and treating the data with cell similarity scores higher than the set threshold as the data of the same cell, dynamically splicing and merging them into a complete data structure; the structured data includes visitor information, reception information and survey content.

[0018] A further technical solution is: in step 2, using Sentence-BERT to calculate the cell similarity score, and setting the threshold to 0.85.

[0019] A further technical solution is that: in step 3, the cross-modal attention mechanism includes dynamic gated channel control and domain adaptive suppression strategy, and the multimodal mask pre-training task includes a cross-modal joint mask strategy and a multimodal contrast alignment mechanism.

[0020] A further technical solution is that: in step 3, the cross-modal joint mask strategy includes a synchronous mask mechanism and sparse guided selection, and the multimodal contrast alignment mechanism includes vector quantization shared space and bidirectional contrast constraints.

[0021] A further technical solution is that in step 3, the hierarchical fine-tuning includes a parameter freezing stage and a semantic enhancement stage, the adversarial training includes inputting noise data and parsing it, calculating the loss between the output of the layout self-attention enhancement model and the true label, and optimizing the layout self-attention enhancement model parameters through back propagation.

[0022] A further technical solution is that: in step 3, the parameter freezing stage includes freezing the weight parameters of all Transformer layers, and only fine-tuning the output classification layer to make the output classification layer adapt to basic entity label recognition, and the basic entity label includes fixed fields for visiting institutions and survey questions; the semantic enhancement stage includes unfreezing the Transformer structure layer parameters, and through optimized training of financial field terms, associating cross-modal semantics, that is, binding the associated information in text, tables, and charts into structured data; generating hierarchical data, that is, building hierarchical relationships based on semantic understanding.

[0023] A further technical solution is that in step 4, the modular information includes the visiting organization and personnel, the receiving personnel and personnel positions, and the research content, and whether the output module is complete is calculated based on the modular information.

[0024] A further technical solution is: it also includes step 5, step 5: modular data management is performed on the structured data obtained in step 2 and the modular information obtained in step 4 to obtain the parsing confidence, the completeness of the key module fields, the noise interference level and the quality of the cross-page table continuation, and the scoring mechanism is started to obtain the total score.

[0025] A further technical solution is: it also includes step 6, step 6: when the total score is ≥85 points, it is directly put into storage; when the total score is <85 points, it is manually reviewed.

[0026] The beneficial effects of adopting the above technical solution are:

[0027] A multimodal parsing method for PDFs used in institutional research projects includes the following steps: Step 1: Parsing a PDF file based on a layout segmentation model to obtain structured and unstructured regions. Structured regions include tables, headers, and footers, while unstructured regions include free text. Step 2: Using a cross-page table reorganization algorithm to process tables in the structured regions, the method extracts key fields from the table data using a rule-based template matching algorithm to form structured data. Step 3: Using a layout self-attention enhancement model, the method includes a sequentially connected input layer, a Transformer encoding layer, and a pre-training task module. The pre-training phase utilizes a cross-modal attention mechanism and a multimodal mask pre-training task, while the fine-tuning phase utilizes layer-by-layer fine-tuning and adversarial training to obtain the layout self-attention enhancement model. Step 4: Using the layout self-attention enhancement model to process unstructured regions, the method obtains modular information. By utilizing the layout self-attention enhancement model, the method achieves high efficiency, good integrity, and high accuracy in PDF file parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the present invention;

[0029] Figure 2 This is the flowchart for step 3;

[0030] Figure 3 This is the principle block diagram of the layout self-attention enhancement model. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0033] Example 1:

[0034] like Figure 1 As shown, the present invention discloses a multimodal parsing method for institutional research PDF, comprising the following steps.

[0035] Step 1: Parse the PDF file.

[0036] Input: Institutional research PDF document.

[0037] Processing algorithm: layout segmentation model.

[0038] Output: structured areas and unstructured areas. Structured areas include tables, headers, and footers, and unstructured areas include free text.

[0039] The layout segmentation model is a deep learning-based computer vision technology that is specifically designed to segment the page content of a PDF document into multiple semantically independent regions, such as text paragraphs, tables, images, formulas, headers, and footers, and accurately locate the boundary coordinates and hierarchical relationships of these elements. This helps locate the core components and accelerate the PDF parsing process.

[0040] Step 2: Structural area processing and cross-page table reorganization.

[0041] Input: The structured region identified in step 1.

[0042] Processing algorithm: cross-page table reorganization algorithm + rule template matching algorithm.

[0043] Output: Complete table data dynamically spliced together. For the table format, keyword fields are extracted through rule matching to obtain visitor information, reception information, and research content in the structured institutional research PDF.

[0044] Cross_tables_score: Sentence-BERT is used to calculate the semantic similarity between the cells above and below the break, with a range of 0.0 to 1.0.

[0045] Specifically, structured area processing extracts key fields such as the visited organization and survey time through rule template matching; in addition, regarding the table part, this application also adds a cross-page table reorganization module to solve the problem of data breakage and ensure the integrity and logic of the content; the specific operation is first to detect the breakpoint; secondly, semantic scoring, using Sentence-BERT to calculate the cell similarity score, according to the format and sentence characteristics of the institutional survey report, adjust the threshold to 0.85, and those above 0.85 are considered to be the same cell, and are dynamically spliced and merged into a complete data structure. The semantics of the Sentence-BERT is BERT, which can be used to quickly evaluate the semantic similarity of cell content.

[0046] Step 3: Build a layout self-attention enhancement model.

[0047] like Figure 3 As shown in Figure 1, the layout self-attention enhancement model includes an input layer, a Transformer encoding layer, and a pre-training task module connected in sequence.

[0048] like Figure 2 As shown in Figure 2, the layout self-attention enhancement model proposed in this application is based on the LayoutLMv3 architecture and is obtained by improving the training strategies in the pre-training and fine-tuning phases. It can more efficiently parse PDF documents related to institutional research. The specific improvement steps are as follows.

[0049] Step 3.1: Pre-training stage improvements.

[0050] During pre-training on a large-scale general document dataset and a self-built financial document dataset, including over 100,000 institutional research PDFs, the LayoutLMv3 model's cross-modal attention mechanism and mask prediction tasks were optimized for financial adaptability by improving the attention mechanism architecture and strengthening multimodal mask pre-training. The specific steps involved are as follows.

[0051] Step 3.1.1: Improve the cross-modal attention mechanism.

[0052] The traditional cross-modal attention mechanism performs redundant calculations on non-critical areas, such as company introductions, resulting in inefficient parsing of long financial documents.

[0053] The innovation of the technical solution of this application is to improve the cross-modal attention mechanism, including: dynamic gated channel control and domain adaptive inhibition strategy. The dynamic gated channel control is to selectively open the interaction path, and the domain adaptive inhibition strategy is to automatically close the redundant module channel.

[0054] Dynamic gating channel control: A trainable gating module is introduced into the cross-modal interaction layer. By analyzing the semantic relevance of text, layout, and image features in real time, the weights for opening each channel are dynamically generated. This reduces the computational effort for image-text association channels in non-critical areas, such as company introductions and disclaimers, by 38%. It also increases the average cross-modal attention weight in core information areas, such as visiting organizations and research content, by 2.7 times.

[0055] Domain adaptive suppression strategy: Furthermore, a high-frequency redundant content template library of institutional research documents is pre-built. When the similarity between the input content and the template library exceeds a threshold, a cross-modal channel closing instruction is automatically triggered. The high-frequency redundant content template library covers eight types of fixed modules, such as company profiles, risk statements, etc.; thereby, the channel activation rate of non-core modules is suppressed from 78% to below 5%, thereby avoiding manual rule maintenance and achieving the technical effect of adaptive migration of domain knowledge.

[0056] Step 3.1.2: Enhance the multimodal mask pre-training task.

[0057] Traditional masking strategies process text and image modalities independently, resulting in the model being unable to establish deep associations between text and images, making it difficult to parse cross-modal coupling content in institutional research documents. The image modality is LayoutLMv3, which only supports single-modal masking. The cross-modal coupling content includes financial data text and corresponding charts.

[0058] The innovative points of the technical solution of this application include: cross-modal joint masking strategy and multimodal contrast alignment mechanism.

[0059] The cross-modal joint masking strategy includes a synchronous masking mechanism and sparse-guided selection, which are described in detail below.

[0060] Synchronous masking mechanism: Implement cross-modal synchronous masking on semantically associated regions, forcing the model to jointly predict the masked content through complementary modal cues. The semantically associated regions include the text segment of revenue data and its adjacent line chart area. The complementary modal cues include the numerical units in the text and the trend features in the image.

[0061] Sparse-guided selection: Dynamically selects high-value mask regions based on sparse cross-modal attention weights, resulting in a 1.8x improvement in training efficiency and 95.3% reconstruction accuracy for key fields, including institution names and financial metrics.

[0062] The multimodal contrastive alignment mechanism includes vector quantization shared space and bidirectional contrastive constraints, which are detailed below.

[0063] Vector Quantization Shared Space: Use a discrete visual vocabulary to encode image regions into visual tags, sharing the embedding space with text or layout tags to achieve a unified semantic representation of cross-modal features.

[0064] Bidirectional contrast constraint: A cross-modal contrast loss is applied to the image and text areas of the synchronized mask, allowing the model to automatically associate adjacent text descriptions when parsing stacked charts, for example, locating the corresponding bar chart area from the text of the net profit margin.

[0065] Step 3.2: Fine-tuning stage.

[0066] The fine-tuning stage includes layer-wise fine-tuning and adversarial training.

[0067] Step 3.2.1: Layer-wise fine-tuning.

[0068] Hierarchical fine-tuning includes a parameter freezing stage and a semantic enhancement stage.

[0069] Parameter freezing stage: Freeze the weight parameters of all Transformer layers and only fine-tune the output classification layer to adapt it to basic entity label recognition, including fixed fields such as visiting institutions and survey questions.

[0070] Semantic enhancement stage: Unfreeze the Transformer structural layer parameters of the upper layer of the model, and achieve cross-modal semantic association through specialized optimization training of financial terminology: bind the related information in text, tables, and charts into structured data. The structured data is not parsed independently; achieve hierarchical data generation: build hierarchical relationships based on semantic understanding.

[0071] The data examples are as follows.

[0072] Consider the input sentence "The company's ROE performance is excellent, at 15%." The fine-tuned model establishes a semantic association between ROE and 15%, while also supporting collaborative parsing of text paragraphs, table cells, chart elements, and multimodal data, ultimately generating a unified structured output: {financial indicator:{ROE:15%}}.

[0073] The text paragraph is: ROE in 2023 is 15%.

[0074] The table unit is: ROE (%)|15.

[0075] The chart element is: a bar chart marking the ROE value of 15%.

[0076] Step 3.2.2: Adversarial training.

[0077] Adding noisy data improves robustness. This noisy data includes blurry scans and handwritten annotations. Of the over 100,000 institutional research documents, 2% are blurry scans with handwritten annotations. For these documents, this application adds adversarial training, giving the model stronger and more comprehensive parsing capabilities. The adversarial training steps are as follows.

[0078] 1. Input the noisy data into the improved LayoutLMv3 model for parsing.

[0079] 2. Calculate the loss between the model output and the true label.

[0080] 3. Optimize model parameters through back propagation to enhance its adaptability to noisy documents.

[0081] After adversarial training, the parsing error rate for blurry scans and files containing handwritten annotations was reduced by 5%.

[0082] Step 4: Unstructured region parsing.

[0083] The unstructured region is input into the layout self-attention enhancement model, and the modular information output includes the visiting organization and personnel, the reception personnel and their positions, and the research content. The completeness of the output module is also calculated.

[0084] Input: Unstructured area.

[0085] Processing algorithm: layout self-attention enhancement model.

[0086] Output: Modularized representation of visiting organizations and individuals, reception personnel and their positions, and research content Q&A. Parsing confidence (PDF_confidence): The average confidence level of key fields based on the model output, used to assess parsing reliability. Noise interference level (Noise_level): Confirms whether the scan is blurry. Key module field completeness (Is_lack): Calculates whether the three modules (visit, reception, and research) are missing.

[0087] Step 5: The system performs modular data management on visiting organizations, personnel information, and survey content, and activates a scoring mechanism. The personnel information includes the reception personnel and their positions.

[0088] Input: parsing confidence, key module field completeness, noise interference level, and cross-page table continuation quality.

[0089] Processing algorithm: Scoring mechanism, see Table 1 for details.

[0090] Output: total score.

[0091] The specific process is: establish a scoring mechanism; the scoring criteria are based on four dimensions: analysis confidence, whether key module fields have content, noise interference level, and cross-page table continuation quality.

[0092] Table 1: Analysis effect score table PDF_SCORE

[0093]

[0094]

[0095] Formula: Total score = ∑(dimension score × weight).

[0096] Step 6: Scoring-based hierarchical review mechanism and efficiency optimization.

[0097] Total score ≥85 points, directly stored in the database: only documents with extremely high scores are allowed to be automatically stored in the database.

[0098] Total score <85 points, manual review: The complete data package is automatically pushed to the manual review platform and risk dimensions are marked. Risk dimensions include missing information and very low confidence in key fields.

[0099] Manual review relies on standardized module data, which increases review efficiency by 95%. The standardized module data includes institutional information, personnel information, research reports and reception records.

[0100] The audit results and modification suggestions are synchronized to the system in real time, the data status is updated and an audit trail log is generated, and finally the data is entered into the database.

[0101] The technical effects are explained as follows.

[0102] The multimodal PDF parsing algorithm and device for institutional research provided in this application achieves the following breakthrough progress in financial document processing through the collaborative innovation of three core modules: page segmentation, cross-page reorganization, and layout self-attention enhancement.

[0103] 1. Multimodal analysis architecture performance upgrade.

[0104] Parsing speed: The full-process processing speed reaches 120 pages / minute, compared to 80 pages / minute with the original technology, an efficiency improvement of 50%.

[0105] Computational optimization: The algorithm's floating-point operations (FLOPs), a measure of the model's computational complexity, were reduced by 30%, reducing inference time to 65% of the original solution.

[0106] Module collaboration: Structured and unstructured parsing are processed in parallel, increasing system resource utilization by 42%.

[0107] 2. Breakthrough in the accuracy of targeted parsing of financial documents.

[0108] Terminology recognition: The accuracy rate of professional field recognition is 98%, while that of traditional OCR technology is 82%.

[0109] Because traditional parsing methods lack financial semantic understanding and fail to establish connections between text, tables, and charts, the accuracy of professional field recognition is 82%. This application improves the professional terminology recognition rate by 16% through dual optimization in the pre-training stage and enhanced semantics in the fine-tuning stage. The dual optimization includes more than 100,000 document training and a cross-modal joint masking strategy.

[0110] Continuation across pages: The table data completeness rate increased from 85% to 98%.

[0111] Traditional parsing methods, such as simple coordinate matching, cannot solve the semantic continuity judgment of the same table, affecting the parsing quality. This application uses Sentence-BERT to calculate the semantic similarity of the context at the break point, and automatically triggers the splicing action when the score is greater than 0.85. Through a semantic-led splicing strategy, it effectively solves the problem of key information loss caused by the splicing of numerical and text tables, which affects subsequent parsing.

[0112] Noise resistance: Thanks to adversarial training optimization, the error rate of parsing blurred scans was reduced from 18% to 13%.

[0113] Traditional parsing methods have basically no ability to process fuzzy scanned documents. This application uses adversarial training in the fine-tuning stage to specifically enhance the parsing and processing capabilities of this type of document.

[0114] 3. The value of industrial deployment is highlighted.

[0115] Automation rate: Due to the adoption of a technical standard of a score ≥ 85 points, 85% of documents are directly stored in the database, and the amount of manual review is reduced by 80%.

[0116] Improved audit efficiency: The time required for manual document processing was reduced from 8 minutes to 0.4 minutes, increasing efficiency by 95%.

[0117] Data compatibility: The field completeness rate of the output standardized structure package is 100%, and the standardized structure package includes institutional information, personnel information, survey content and financial indicators.

[0118] Practical application verification: A financial platform company's processing efficiency increased fourfold, processing an average of 3,000 documents per month; customer complaints caused by data errors and missing data decreased by 92%.

[0119] Example 2:

[0120] The present invention discloses a multimodal analysis device for an institutional research PDF, which is used for the multimodal analysis method for an institutional research PDF described in Example 1 and includes the following program modules.

[0121] Input module 1: used to input PDF files.

[0122] Transmission module 2: used to input the PDF file into the multimodal PDF parsing module for parsing.

[0123] The multimodal PDF parsing module includes the following program modules.

[0124] Parsing module 3.1: used for page segmentation and distinguishing between structured text and unstructured text.

[0125] Structured region processing module 3.2: used to parse structured text and complete cross-page table reorganization.

[0126] Unstructured Region Parsing Module 3.3: Used to parse unstructured text using the trained layout self-attention enhancement model.

[0127] Scoring module 4: used to score the analysis results. High-scoring ones are directly stored in the database, while low-scoring ones are transferred to the manual review platform for manual review and then stored in the database.

[0128] Technical Effects: Automated parsing of institutional research PDF documents reduces manual intervention. Addresses semantic fragmentation in tables across multiple pages, improving data integrity. Enhanced recognition of financial terminology improves parsing accuracy. A layout self-attention enhancement model is added to unstructured text parsing, enhancing parsing completeness and accuracy.

[0129] Overall, this application parses PDF files more efficiently and accurately, and through fine-tuning, it is more suitable for institutional research documents. It resolves the semantic gap between tables across pages, improving data integrity. Furthermore, a layout self-attention enhancement model is added to the parsing of unstructured text, improving the completeness of parsed tables and the accuracy of specialized fields. These specialized fields include: 1. Visitor information, reception information, and research content in institutional research PDFs; 2. Accurately identifying financial terminology, such as financial indicator terms and compliance and risk control terms.

Claims

1. A multimodal analysis method for PDFs of institutional surveys, characterized by: The following steps are included: Step 1: Parse the PDF file based on the layout segmentation model to obtain structured and unstructured regions. Structured regions include tables, headers, and footers, while unstructured regions include free text. Step 2: The table in the structured area is processed by the cross-page table reorganization algorithm to obtain complete table data. The table data is extracted by the rule template matching algorithm to obtain key fields and form structured data; Step 3: The layout self-attention enhancement model includes a sequentially connected input layer, a Transformer encoding layer, and a pre-training task module. In the pre-training stage, a cross-modal attention mechanism and a multimodal mask pre-training task are used. In the fine-tuning stage, layer-by-layer fine-tuning and adversarial training are used to obtain the layout self-attention enhancement model. Step 4: The unstructured area is processed by the layout self-attention enhancement model to obtain modular information.

2. The multimodal analysis method for PDFs of institutional research according to claim 1 is characterized by: In step 2, the steps of the cross-page table reorganization algorithm processing include breakpoint detection, calculating the cell similarity score based on the format and sentence features of the PDF file, treating the data with cell similarity scores higher than the set threshold as the data of the same cell, dynamically splicing and merging them into a complete data structure; the structured data includes visit information, reception information and survey content.

3. The multimodal analysis method for PDFs of institutional research according to claim 2 is characterized by: In step 2, Sentence-BERT is used to calculate the cell similarity score, and the threshold is set to 0.

85.

4. The multimodal analysis method for PDFs of institutional research according to claim 1 is characterized by: In step 3, the cross-modal attention mechanism includes dynamic gated channel control and domain adaptive suppression strategy, and the multimodal mask pre-training task includes a cross-modal joint mask strategy and a multimodal contrast alignment mechanism.

5. The multimodal analysis method for PDF of institutional research according to claim 4 is characterized by: In step 3, the cross-modal joint masking strategy includes a synchronous masking mechanism and sparse guided selection, and the multimodal contrast alignment mechanism includes vector quantization shared space and bidirectional contrast constraints.

6. The multimodal analysis method for PDF of institutional research according to claim 1, characterized in that: In step 3, the hierarchical fine-tuning includes a parameter freezing stage and a semantic enhancement stage, and the adversarial training includes inputting noise data and parsing it, calculating the loss between the output of the layout self-attention enhancement model and the true label, and optimizing the layout self-attention enhancement model parameters through back propagation.

7. The multimodal analysis method for PDF of institutional research according to claim 6, characterized in that: In step 3, the parameter freezing stage includes freezing the weight parameters of all Transformer layers and only fine-tuning the output classification layer to adapt the output classification layer to basic entity label recognition. The basic entity label includes fixed fields for the visiting organization and the survey question. The semantic enhancement stage includes unfreezing the Transformer structural layer parameters, associating cross-modal semantics through optimized training of financial field terminology, that is, binding the related information in text, tables, and charts into structured data; generating hierarchical data, that is, building hierarchical relationships based on semantic understanding.

8. The multimodal analysis method for PDF of institutional research according to claim 1 is characterized by: In step 4, the modular information includes the visiting organization and personnel, the receiving personnel and personnel positions, and the research content, and whether the output module is complete is calculated based on the modular information.

9. The multimodal analysis method for PDF of institutional research according to claim 1, characterized in that: It also includes step 5, step 5: modular data management is performed on the structured data obtained in step 2 and the modular information obtained in step 4 to obtain the parsing confidence, the completeness of the key module fields, the noise interference level and the quality of the cross-page table continuation, and the scoring mechanism is started to obtain the total score.

10. The multimodal analysis method for PDF of institutional research according to claim 9, characterized in that: It also includes step 6. Step 6: When the total score is ≥85 points, it will be directly put into storage; when the total score is <85 points, it will be manually reviewed.

Images