A multi-modal analysis method for institutional survey PDFs

Abstract

The application discloses a kind of multi-modal analysis methods for mechanism investigation PDF, it is related to electric digital data processing technical field;It includes the following steps, step 1: based on layout segmentation model analysis PDF file obtains structured area and unstructured area;Step 2: the table of structured area is handled by cross-page table reorganization algorithm to obtain complete table data, and key field is extracted by rule template matching algorithm to form structured data;Step 3: layout self-attention enhancement model includes sequentially connected input layer, Transform coding layer and pre-training task module, cross-modal attention mechanism and multi-modal mask pre-training task are used in pre-training stage, and hierarchical fine-tuning and adversarial training are used in fine-tuning stage;Step 4: unstructured area is handled by layout self-attention enhancement model to obtain modular information;It realizes high analysis efficiency, good integrity and good accuracy by layout self-attention enhancement model etc..

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a multimodal parsing method for institutional survey PDFs. Background Technology

[0002] Currently, during the research process, a PDF document detailing the institutional research is released, containing information such as the research methods, timeline, institution, host personnel, and research content. The research content typically employs a question-and-answer format. Existing technology usually relies on traditional Optical Character Recognition (OCR) technology to parse PDF files, and key information is extracted and recorded through manual reading and review.

[0003] The application, with publication number CN118747902A, is titled "A Method and System for Table Text Recognition Based on OCR," and includes: parsing PDF files and converting them into image data; performing layout analysis on the image data to obtain the layout analysis results. It employs traditional parsing methods, such as PyPDF2 and Optical Character Recognition (OCR) technology, to directly extract text objects from the content stream. However, due to its reliance on font encoding mapping to solve character encoding problems, it often faces multiple limitations in application, leading to incomplete parsing of complex elements such as tables and images, difficulties in continuing content across pages, low parsing accuracy, and 90% reliance on manual review.

[0004] The application, published under CN119360398A, is titled "A Method for Efficient Semantic Understanding of PDF Text Using Deep Learning." It includes: parsing multimodal PDF data from the construction industry, learning modal structured information, and constructing a segmentation and extraction module. This module is used to parse the PDF modal distribution and extract structured information. However, the PDF parsing speed is slow, and invalid information consumes parsing time. It also relies excessively on manual review, resulting in low efficiency.

[0005] Based on the analysis of the two documents mentioned above and the existing technical solutions, the existing technical solutions have the following technical problems.

[0006] The limitations of traditional optical character recognition (OCR) technology: It relies on a single rule or general model and cannot distinguish between structured and unstructured regions, resulting in information fragmentation. Structured regions include tables and headers, while unstructured regions include free text.

[0007] Issues with cross-page table processing: Existing solutions, such as coordinate matching, cannot solve the semantic breakage problem, leading to the loss of critical data.

[0008] Poor domain adaptability: General deep learning natural language processing models have low recognition rates for financial terms such as operating revenue and inventory turnover.

[0009] Inefficient: Manual review requires checking each field one by one, which is time-consuming and prone to errors. Summary of the Invention

[0010] This invention provides a multimodal parsing method for PDFs used in institutional research, which solves the technical problems of low parsing efficiency, poor integrity, and poor accuracy in PDF files.

[0011] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0012] A multimodal parsing method for institutional survey PDFs includes the following steps:

[0013] Step 1: Parse the PDF file based on the page layout model to obtain structured and unstructured areas. Structured areas include tables, headers, and footers, while unstructured areas include free text.

[0014] Step 2: The structured area tables are processed by a cross-page table reorganization algorithm to obtain complete table data. The table data is then processed by a rule template matching algorithm to extract key fields and form structured data.

[0015] Step 3: The layout self-attention enhancement model consists of an input layer, a Transformer encoding layer, and a pre-training task module connected in sequence. In the pre-training stage, a cross-modal attention mechanism and a multimodal mask pre-training task are used. In the fine-tuning stage, hierarchical fine-tuning and adversarial training are used to obtain the layout self-attention enhancement model.

[0016] Step 4: Unstructured regions are processed by a layout self-attention enhancement model to obtain modular information.

[0017] A further technical solution is as follows: In step 2, the steps of the cross-page table reorganization algorithm include breakpoint detection, calculating cell similarity scores based on the format and sentence features of the PDF file, and dynamically splicing and merging data with cell similarity scores higher than a set threshold into data of the same cell to form a complete data structure; the structured data includes visitor information, reception information, and survey content.

[0018] A further technical solution is to use Sentence-BERT to calculate cell similarity scores in step 2, and set a threshold of 0.85.

[0019] A further technical solution is that, in step 3, the cross-modal attention mechanism includes dynamic gated channel control and a domain-adaptive inhibition strategy, and the multimodal mask pre-training task includes a cross-modal joint masking strategy and a multimodal contrast alignment mechanism.

[0020] A further technical solution is that, in step 3, the cross-modal joint masking strategy includes a synchronization masking mechanism and sparse guided selection, and the multimodal contrast alignment mechanism includes vector quantization shared space and bidirectional contrast constraint.

[0021] A further technical solution is as follows: In step 3, the hierarchical fine-tuning includes a parameter freezing stage and a semantic enhancement stage, and the adversarial training includes inputting noisy data and parsing it, calculating the loss between the output of the layout self-attention enhancement model and the real label, and optimizing the parameters of the layout self-attention enhancement model through backpropagation.

[0022] A further technical solution is as follows: In step 3, the parameter freezing stage includes freezing the weight parameters of all Transformer layers and fine-tuning the training of only the output classification layer to make the output classification layer adapt to the recognition of basic entity labels. The basic entity labels include fixed fields of visiting institutions and survey questions. The semantic enhancement stage includes unfreezing the Transformer structure layer parameters, optimizing the training through financial terminology, associating cross-modal semantics, that is, binding the associated information in text, tables, and charts into structured data; generating hierarchical data, that is, constructing hierarchical relationships based on semantic understanding.

[0023] A further technical solution is as follows: In step 4, the modular information includes the visiting organizations and personnel, the receiving personnel and their positions, and the survey content. The completeness of the output module is calculated based on the modular information.

[0024] A further technical solution includes step 5, which involves performing modular data management on the structured data obtained in step 2 and the modular information obtained in step 4 to obtain the resolution confidence, key module field completeness, noise interference level, and cross-page table continuation quality, and then initiating a scoring mechanism to obtain the total score.

[0025] A further technical solution includes step 6: when the total score is ≥85 points, it is directly entered into the database; when the total score is <85 points, it is manually reviewed.

[0026] The beneficial effects of adopting the above technical solution are as follows:

[0027] A multimodal parsing method for PDFs used in institutional research includes the following steps: Step 1: Parsing the PDF file based on a layout segmentation model to obtain structured and unstructured regions. The structured regions include tables, headers, and footers, while the unstructured regions include free text. Step 2: The tables in the structured regions are processed by a cross-page table recombination algorithm to obtain complete table data. The table data is then extracted using a rule template matching algorithm to obtain key fields and form structured data. Step 3: A layout self-attention enhancement model includes a sequentially connected input layer, a Transformer encoding layer, and a pre-training task module. During the pre-training stage, a cross-modal attention mechanism and a multimodal mask pre-training task are used. During the fine-tuning stage, hierarchical fine-tuning and adversarial training are used to obtain the layout self-attention enhancement model. Step 4: The unstructured regions are processed by the layout self-attention enhancement model to obtain modular information. Through the layout self-attention enhancement model and other methods, this method achieves high efficiency, good integrity, and high accuracy in PDF file parsing. Attached Figure Description

[0028] Figure 1 This is a flowchart of the present invention;

[0029] Figure 2 This is the flowchart for step 3;

[0030] Figure 3 This is a schematic diagram of the principle of the layout self-attention enhancement model. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this application or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0032] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0033] Example 1:

[0034] like Figure 1 As shown, this invention discloses a multimodal parsing method for institutional survey PDFs, which includes the following steps.

[0035] Step 1: Parse the PDF file.

[0036] Input: Institutional survey PDF document.

[0037] Processing algorithm: Page segmentation model.

[0038] Output: Structured and unstructured regions. Structured regions include tables, headers, and footers, while unstructured regions include free text.

[0039] Page layout segmentation is a deep learning-based computer vision technology specifically designed to segment the content of a PDF document into multiple semantically independent regions, such as text paragraphs, tables, images, formulas, headers, and footers, and accurately locate the boundary coordinates and hierarchical relationships of these elements. This helps to pinpoint the core components and accelerate the PDF parsing process.

[0040] Step 2: Structured area processing, cross-page table reorganization.

[0041] Input: The structured regions identified in step 1.

[0042] Processing algorithm: cross-page table reorganization algorithm + rule template matching algorithm.

[0043] Output: A dynamically assembled, complete table of data. For the table format, keyword fields are extracted using rule matching to obtain structured information from the organization's survey PDF, including visitor details, reception information, and survey content.

[0044] Cross-tables score: The semantic similarity between the cells above and below the breakpoint is calculated using Sentence-BERT, ranging from 0.0 to 1.0.

[0045] Specifically, structured area processing extracts key fields, such as visiting institutions and survey time, through rule template matching. Regarding the table section, this application also incorporates a cross-page table reorganization module to address data breakage issues and ensure content integrity and logical consistency. The specific operation involves first detecting breakpoints; second, semantic scoring, using Sentence-BERT to calculate cell similarity scores. Based on the format and sentence characteristics of the institution survey report, a threshold of 0.85 is adjusted. Cells with similarities higher than 0.85 are considered to be from the same cell and are dynamically concatenated to form a complete data structure. The semantics of Sentence-BERT are based on BERT, which can be used to quickly assess the semantic similarity of cell content.

[0046] Step 3: Construct a layout-based self-attention enhancement model.

[0047] like Figure 3 As shown, the layout self-attention enhancement model consists of an input layer, a Transformer encoding layer, and a pre-trained task module connected in sequence.

[0048] like Figure 2 As shown, the layout self-attention enhancement model proposed in this application is obtained by improving the training strategy in the pre-training and fine-tuning stages based on the LayoutLMv3 architecture, which can more efficiently parse PDF documents related to institutional surveys. The specific improvement steps are as follows.

[0049] Step 3.1: Improvements to the pre-training phase.

[0050] In the pre-training process of integrating a large-scale general document dataset and a self-built financial document dataset, adaptive optimization for the financial domain is achieved by improving the attention mechanism architecture and strengthening multimodal mask pre-training for the cross-modal attention mechanism and mask prediction task of the LayoutLMv3 model. The self-built financial document dataset includes more than 100,000 institutional survey PDFs. The specific steps are as follows.

[0051] Step 3.1.1: Improve cross-modal attention mechanism.

[0052] Traditional cross-modal attention mechanisms perform redundant calculations on non-critical regions, resulting in low parsing efficiency for long financial documents, where the non-critical regions are company introductions.

[0053] The innovation of this application's technical solution lies in the improved cross-modal attention mechanism, which includes: dynamic gating channel control and a domain-adaptive inhibition strategy. The dynamic gating channel control selectively opens the interaction path, and the domain-adaptive inhibition strategy automatically closes redundant module channels.

[0054] Dynamic gated channel control: A trainable gating module is introduced into the cross-modal interaction layer. By analyzing the semantic correlation of text, layout, and image features in real time, the opening weight of each channel is dynamically generated, which reduces the computation of image-text association channels in non-critical areas by 38%. The non-critical areas include company introduction and disclaimer. The average cross-modal attention weight of core information areas is increased by 2.7 times. The core information areas include visiting organizations and survey content.

[0055] Domain-adaptive suppression strategy: Further, a high-frequency redundant content template library for institutional research documents is pre-built. When the similarity between the input content and the template library exceeds a threshold, a cross-modal channel closure instruction is automatically triggered. The high-frequency redundant content template library covers eight fixed modules, such as company profiles and risk statements. As a result, the channel activation rate of non-core modules is suppressed from 78% to below 5%. This avoids manual rule maintenance and achieves the technical effect of adaptive transfer of domain knowledge.

[0056] Step 3.1.2: Enhance the multimodal mask pre-training task.

[0057] Traditional masking strategies process text and image modalities independently, making it difficult for the model to establish a deep relationship between text and images and to parse cross-modal coupled content in institutional research documents. The image modality is LayoutLMv3, which only supports single-modal masking. The cross-modal coupled content includes financial data text and corresponding charts.

[0058] The innovative aspects of this application's technical solution include a cross-modal joint masking strategy and a multi-modal comparison and alignment mechanism.

[0059] The cross-modal joint masking strategy includes a synchronization masking mechanism and sparse guided selection, which are detailed below.

[0060] Synchronous masking mechanism: Cross-modal synchronous masking is implemented on semantically related regions, forcing the model to jointly predict the masked content through complementary modal cues. The semantically related regions include text segments of revenue data and their adjacent line graph regions. The complementary modal cues include numerical units in the text and trend features in the image.

[0061] Sparse-guided selection: Based on sparse cross-modal attention weights, high-value mask regions are dynamically selected, which improves training efficiency by 1.8 times and achieves a reconstruction accuracy of 95.3% for key fields, including organization name and financial indicators.

[0062] The multimodal contrast alignment mechanism includes a vector quantization shared space and bidirectional contrast constraints, which are detailed below.

[0063] Vector quantization shared space: Image regions are encoded into visual tags using a discrete visual vocabulary, which share the embedding space with text or layout tags to achieve a unified semantic representation of cross-modal features.

[0064] Two-way contrast constraint: Apply cross-modal contrast loss to the text and image regions of the synchronization mask, so that when the model parses stacked charts, it automatically associates adjacent text descriptions, such as locating the corresponding bar chart region from the text of net profit margin.

[0065] Step 3.2: Fine-tuning stage.

[0066] The fine-tuning phase includes tiered fine-tuning and adversarial training.

[0067] Step 3.2.1: Layered fine-tuning.

[0068] Layered fine-tuning includes a parameter freezing phase and a semantic enhancement phase.

[0069] Parameter freezing phase: Freeze the weight parameters of all Transformer layers and fine-tune the training only for the output classification layer to make the output classification layer adapt to basic entity label recognition, including fixed fields such as visiting organizations and survey questions.

[0070] Semantic enhancement stage: Unfreeze the parameters of the Transformer structure layer in the upper layer of the model, and achieve cross-modal semantic association through specialized optimization training of financial terminology: bind the related information in text, tables and charts into structured data, and the structured data is not parsed independently; realize hierarchical data generation: construct hierarchical relationships based on semantic understanding.

[0071] The data example is as follows.

[0072] Input statement: The company's ROE performance is excellent, with a value of 15%. The fine-tuned model can establish a semantic association between ROE and 15%; it also supports collaborative parsing of text paragraphs, table cells, chart elements, and multimodal data, ultimately generating a unified structured output: {Financial Indicator:{ROE:15%}}.

[0073] The text paragraph states: "The ROE in 2023 was 15%."

[0074] The table cell is: ROE(%)|15.

[0075] The chart element is a bar chart indicating the ROE value of 15%.

[0076] Step 3.2.2: Adversarial training.

[0077] To improve robustness, noisy data was added, including blurred scans and handwritten annotations. Of the more than 100,000 institutional survey documents, 2% were blurred scans with handwritten annotations. For these texts, this application added adversarial training to give the model a stronger and more comprehensive analytical capability. The adversarial training steps are as follows.

[0078] 1. Input the noise data into the improved LayoutLMv3 model and parse it.

[0079] 2. Calculate the loss between the model output and the true label.

[0080] 3. Optimize model parameters through backpropagation to enhance its adaptability to noisy documents.

[0081] After adversarial training, the parsing error rate for blurry scanned documents and documents containing handwritten annotations decreased by 5%.

[0082] Step 4: Unstructured region resolution.

[0083] Input an unstructured region into a layout-based self-attention enhancement model, and output modular information including visiting organizations and personnel, reception staff and their positions, and survey content. Also, calculate whether the output modules are complete.

[0084] Input: Unstructured region.

[0085] Processing algorithm: Layout self-attention enhancement model.

[0086] Output: Modularized lists of visiting organizations and personnel, reception staff and their positions, and survey questions and answers. PDF confidence: Average confidence score based on key fields in the model output, used to assess parsing reliability. Noise level: Confirms whether the document is a blurry scan. Key module field completeness (Is_lack): Calculates whether any of the three modules (visitors, reception staff, and survey participants) are missing.

[0087] Step 5: The system performs modular data management of visiting organizations, personnel information, and survey content, and initiates a scoring mechanism. The personnel information includes the personnel who received the visitor and their positions.

[0088] Input: Confidence level of parsing, completeness of key module fields, noise interference level, and quality of cross-page table continuation.

[0089] Processing algorithm: scoring mechanism, see Table 1 for details.

[0090] Output: Total score.

[0091] The specific process is as follows: establish a scoring mechanism; the scoring criteria are based on four dimensions: parsing confidence, whether key module fields have content, noise interference level, and cross-page table continuation quality.

[0092] Table 1: Analysis Performance Rating Table PDF_SCORE

[0093]

[0094]

[0095] The formula for total score is: Total Score = ∑(Dimensional Score × Weight).

[0096] Step 6: Rating-based tiered review mechanism and efficiency optimization.

[0097] Documents with a total score of 85 or higher will be directly added to the database: Only documents with extremely high scores are allowed to be automatically added to the database.

[0098] If the total score is less than 85 points, manual review will be conducted: The complete data package will be automatically pushed to the manual review platform, and risk dimensions will be marked. Risk dimensions include missing information, very low confidence of key fields, etc.

[0099] Manual review relies on standardized module data, which improves review efficiency by 95%. The standardized module data includes organizational information, personnel information, research reports, and reception records.

[0100] The review results and modification suggestions are synchronized to the system in real time, the data status is updated and an review trajectory log is generated, and finally the data is entered into the database.

[0101] The technical effects are explained below.

[0102] The multimodal PDF parsing algorithm and device for institutional research provided in this application achieve the following breakthroughs in financial document processing through the synergistic innovation of three core modules: page segmentation, cross-page reorganization, and layout self-attention enhancement.

[0103] 1. Performance upgrade of multimodal parsing architecture.

[0104] Parsing speed: The entire process speed reaches 120 pages per minute, compared to 80 pages per minute for the original technology, resulting in a 50% improvement in efficiency.

[0105] Computational optimization: The number of floating-point operations (FLOPs) in the algorithm is reduced by 30%, and the inference time is reduced to 65% of the original solution.

[0106] Module collaboration: Parallel processing of structured and unstructured parsing improves system resource utilization by 42%.

[0107] 2. Breakthrough in the precision of targeted parsing of financial documents.

[0108] Terminology recognition: 98% accuracy in recognizing specialized fields, compared to 82% for traditional OCR technology.

[0109] Because traditional parsing methods lack financial semantic understanding and fail to establish connections between text, tables, and charts, the accuracy of professional field recognition is only 82%. This application improves the professional term recognition rate by 16% through dual optimization in the pre-training stage and semantic enhancement in the fine-tuning stage. The dual optimization includes training on more than 100,000 documents and a cross-modal joint masking strategy.

[0110] Cross-page continuation: Table data integrity rate increased from 85% to 98%.

[0111] Traditional parsing methods that rely solely on coordinate matching cannot determine the semantic continuity of tables within the same table, thus affecting parsing quality. This application uses Sentence-BERT to calculate the semantic similarity of the context at the break point; a score greater than 0.85 automatically triggers a splicing action. Through a semantically driven continuation strategy, it effectively solves the problem of key information loss that occurs when continuating numerical and textual tables, impacting subsequent parsing.

[0112] Noise resistance: Thanks to the use of adversarial training optimization, the resolution error rate of blurred scans has been reduced from 18% to 13%.

[0113] Traditional parsing methods have virtually no ability to process fuzzy scanned documents. This application enhances the parsing and processing capabilities for this type of document through adversarial training during the fine-tuning phase.

[0114] 3. The value of industrial deployment is becoming increasingly apparent.

[0115] Automation rate: Due to the adoption of a technical standard with a score of ≥85, 85% of documents are directly entered into the database, reducing the amount of manual review by 80%.

[0116] Improved efficiency in review: Manual processing time for a single document was reduced from 8 minutes to 0.4 minutes, increasing efficiency by 95%.

[0117] Data compatibility: The output standardized structure package has 100% field completeness, including organizational information, personnel information, survey content, and financial indicators.

[0118] Real-world application verification: A financial platform company saw a fourfold increase in processing efficiency, handling an average of 3,000 documents per month; customer complaints due to data errors and missing data decreased by 92%.

[0119] Example 2:

[0120] This invention discloses a multimodal parsing device for institutional survey PDFs, used in the multimodal parsing method for institutional survey PDFs described in Example 1, and includes the following program modules.

[0121] Input module 1: Used for inputting PDF files.

[0122] Transmission module 2: Used to input PDF files into the multimodal PDF parsing module for parsing.

[0123] The multimodal PDF parsing module includes the following program modules.

[0124] Parsing module 3.1: Used for page layout cutting, distinguishing between structured and unstructured text.

[0125] Structured Region Processing Module 3.2: Used to parse structured text and reorganize cross-page tables.

[0126] Unstructured Region Parsing Module 3.3: Used to parse unstructured text using a trained layout self-attention enhancement model.

[0127] Scoring Module 4: This module scores the parsed results. High-scoring results are directly entered into the database, while low-scoring results are sent to the manual review platform for manual review before being entered into the database.

[0128] Technical Benefits: Enables automated parsing of institutional research PDF documents, reducing manual intervention. Resolves semantic breaks in cross-page tables, improving data integrity. Enhances the recognition of financial terminology, improving parsing accuracy. Incorporates a layout-based self-attention enhancement model into unstructured text parsing, further improving parsing completeness and accuracy.

[0129] In summary, this application parses PDF files more efficiently and accurately, and through fine-tuning, makes it more suitable for institutional research documents. It resolves the semantic breakage issue in cross-page tables, improving data integrity. Furthermore, it incorporates a layout self-attention enhancement model into unstructured text parsing, improving the completeness of parsed tables and the accuracy of specialized fields. These specialized fields include: 1. Visitor information, reception information, and research content in institutional research PDFs; 2. Accurate identification of financial terminology, such as financial indicator terms and compliance and risk control terms.

Claims

1. A multimodal parsing method for institutional survey PDFs, characterized in that: Includes the following steps, Step 1: Parse the PDF file based on the page layout model to obtain structured and unstructured areas. Structured areas include tables, headers, and footers, while unstructured areas include free text. Step 2: The structured area tables are processed by a cross-page table reorganization algorithm to obtain complete table data. The table data is then processed by a rule template matching algorithm to extract key fields and form structured data. Step 3: The layout self-attention enhancement model comprises an input layer, a Transformer encoding layer, and a pre-training task module connected in sequence. During the pre-training phase, a cross-modal attention mechanism and a multimodal mask pre-training task are employed. In the fine-tuning phase, hierarchical fine-tuning and adversarial training are used to obtain the layout self-attention enhancement model. The cross-modal attention mechanism includes dynamic gating channel control and a domain-adaptive inhibition strategy. The dynamic gating channel control introduces a trainable gating module to analyze the semantic correlation of text, layout, and image features in real time, dynamically generating the opening weights for each channel. The domain-adaptive inhibition strategy uses a pre-built high-frequency redundant content template library of institutional survey documents. When the similarity between the input content and the template library exceeds a threshold, a cross-modal channel closing instruction is automatically triggered. The multimodal mask pre-training task... The system includes a cross-modal joint masking strategy and a multimodal contrast alignment mechanism. The cross-modal joint masking strategy comprises a synchronous masking mechanism and sparse guided selection. The synchronous masking mechanism applies cross-modal synchronous masking to semantically related regions, forcing the model to jointly predict the masked content using complementary modal cues. The sparse guided selection dynamically selects high-value masked regions based on sparse cross-modal attention weights. The multimodal contrast alignment mechanism includes a vector quantization shared space and a bidirectional contrast constraint. The vector quantization shared space uses a discrete visual vocabulary to encode image regions into visual markers, sharing an embedding space with text or layout markers to achieve a unified semantic representation of cross-modal features. The bidirectional contrast constraint applies a cross-modal contrast loss to the image and text regions of the synchronous mask, enabling the model to automatically associate adjacent text descriptions when parsing stacked charts. Step 4: Unstructured regions are processed by a layout self-attention enhancement model to obtain modular information.

2. The multimodal parsing method for institutional survey PDFs according to claim 1, characterized in that: In step 2, the cross-page table reorganization algorithm processes the following steps: breakpoint detection, calculating cell similarity scores based on the format and sentence features of the PDF file, and dynamically splicing and merging data with cell similarity scores higher than a set threshold into a complete data structure. The structured data includes visitor information, reception information, and survey content.

3. The multimodal parsing method for institutional survey PDFs according to claim 2, characterized in that: In step 2, Sentence-BERT is used to calculate cell similarity scores, with a threshold of 0.

85.

4. The multimodal parsing method for institutional survey PDFs according to claim 1, characterized in that: In step 3, the hierarchical fine-tuning includes a parameter freezing stage and a semantic enhancement stage. The adversarial training includes inputting noisy data and parsing it, calculating the loss between the output of the layout self-attention enhancement model and the real label, and optimizing the parameters of the layout self-attention enhancement model through backpropagation.

5. The multimodal parsing method for institutional survey PDFs according to claim 4, characterized in that: In step 3, the parameter freezing stage includes freezing the weight parameters of all Transformer layers and fine-tuning the training of only the output classification layer to make the output classification layer adapt to the basic entity label recognition. The basic entity labels include fixed fields of visiting organizations and survey questions. The semantic enhancement stage includes unfreezing the Transformer structure layer parameters, optimizing training with financial terminology, associating cross-modal semantics (i.e., binding related information in text, tables, and charts into structured data), and generating hierarchical data (i.e., constructing hierarchical relationships based on semantic understanding).

6. The multimodal parsing method for institutional survey PDFs according to claim 1, characterized in that: In step 4, the modular information includes the visiting organizations and personnel, the receiving personnel and their positions, and the survey content. The completeness of the output module is calculated based on the modular information.

7. The multimodal parsing method for institutional survey PDFs according to claim 1, characterized in that: It also includes step 5: performing modular data management on the structured data obtained in step 2 and the modular information obtained in step 4 to obtain the resolution confidence, key module field completeness, noise interference level and cross-page table continuation quality, starting the scoring mechanism, and obtaining the total score.

8. The multimodal parsing method for institutional survey PDFs according to claim 7, characterized in that: It also includes step 6: When the total score is ≥85 points, it is directly entered into the database; when the total score is <85 points, it is manually reviewed.

Images