A document title level optimization method based on multi-modal data processing

Abstract

This invention discloses a document title hierarchy optimization method based on multimodal data processing, belonging to the field of intelligent document processing technology. The method includes: PDF multimodal information collection and cleaning; improving region classification accuracy through multimodal feature injection; aligning text semantics, visual layout, and image association features using the DTW algorithm, and generating fused feature vectors based on attention mechanisms; designing structured prompt templates and setting static rules and dynamic logical constraints to guide large-scale model reasoning; and forming closed-loop optimization through multimodal joint verification. This method improves the accuracy of title hierarchy recognition in complex documents (including charts and multi-column layouts), especially for easily confused categories such as figure titles and table titles. By fusing multimodal features such as font style, page layout, and image association, the hierarchical relationship logic is made more reasonable, thus improving accuracy.

Description

Technical Field

[0001] This invention belongs to the field of intelligent document processing technology, specifically relating to a document title hierarchy optimization method based on multimodal data processing. Background Technology

[0002] With the advent of the digital age, the number of documents (such as textbooks, white papers, and academic papers) has exploded. These documents typically contain complex hierarchical heading structures, and accurately identifying and optimizing these heading levels is crucial for document reading, comprehension, retrieval, and analysis. Meanwhile, against the backdrop of this explosive growth in digital documents, Retrieval Enhancement Generation (RAG), as a core technology connecting massive amounts of documents with precise question answering, relies heavily on the accuracy of retrieval and the degree of knowledge structuring. Accurate identification and optimization of document heading levels provide key support for RAG to address its core pain points.

[0003] Currently, document heading level processing mainly suffers from the following problems: existing layout analysis models lack sufficient accuracy in recognizing document elements, especially in distinguishing heading levels; heading level recognition relies excessively on tag templates, lacking an understanding of the overall logical structure and semantics of the document; and the processing workflow has a low degree of automation, requiring a large amount of manual intervention, resulting in low efficiency.

[0004] Therefore, developing a method that can automatically and accurately optimize document heading levels has become a pressing technical problem in this field. Summary of the Invention

[0005] To overcome the shortcomings and deficiencies of the existing technology, the present invention adopts the following technical solution: A document title hierarchy optimization method based on multimodal data processing includes the following steps: Step S101: Multimodal document data acquisition and preprocessing; Step S102: Input the preprocessed multimodal data into the optimized layout analysis module, convert the font style and page layout features into vectors and concatenate them with the image RGB features, then input them into the model backbone, optimize the coordinates of the title region detection box, and output the layout analysis results of multiple detection boxes and the multimodal features of each title region; Step S103: Use the dynamic time warping algorithm to align the text semantic features, visual layout features and image association features of the title, output an equal-length feature sequence, calculate the weight of each modality feature, and generate a multimodal fusion feature vector by weighted summation; Step S104: Concatenate the initial OCR Markdown text, multimodal feature description, and multimodal fusion feature vector summary into a prompt input and feed it into the QWen2.5VL-72B large model. Enable static rules and dynamic logical constraints to guide the large model's inference and output the preliminary results of the title level in Markdown format. Step S105: Calculate the confidence score of the results by combining text semantic consistency and visual feature consistency through a multimodal joint verification engine. When the confidence score is lower than the preset confidence threshold, trigger manual review, collect error cases found during the review, mark the error cases as training data, and drive the parameter iterative training of the optimized layout analysis module and the QWen2.5VL-72B large model enhancement module.

[0006] Preferably, in step S101, specifically: the text data is converted into a string format, the image data is converted into an RGB channel PNG format with a unified resolution of a preset resolution, and the font style and page layout information are stored as a structured attribute dictionary; the cleaning operation of the cross-source data cleaning submodule includes: deleting duplicate PNG images based on image hash value comparison, removing blank pages by dual determination of text character count and image pixel value, and completing and repairing missing font attribute data based on interpolation of font features of adjacent text in the same document.

[0007] Preferably, in step S102, the multimodal feature injection submodule converts the font style and page layout features into vectors, and then concatenates them with the image RGB features to input the backbone network of the model; the detection box fine adjustment submodule combines the text line boundary and image edge information to automatically expand the height of the long title detection box spanning multiple lines to the full text line range.

[0008] Preferably, in step S103, a multimodal feature extraction step is included before multimodal feature alignment and fusion, specifically: Text semantic features: The title text is encoded using a pre-trained BERT-base model, and the output is a vector with the first preset dimension; Visual layout features: normalize the font size of the title area to the 0-1 range, add bold font markers (b, 0 = not bold, 1 = bold), and normalize the vertical coordinates of the page to the 0-1 range and concatenate them into a vector with a second preset dimension; Image association features: If the title corresponds to an associated image, the ResNet50 model is used to extract image features and reduce the dimensionality to the third preset dimension; if there is no associated image, the zero vector of the third preset dimension is filled.

[0009] Preferably, in step S103, the cross-modal feature alignment engine uses the DTW algorithm for alignment, which includes: alignment error evaluation: calculating dimensional difference error and spatial position error; when the dimensional difference error exceeds a preset dimensional difference threshold, mapping the low-dimensional features to a target dimension consistent with the text semantic feature dimension through a fully connected layer; when the spatial position error exceeds a preset spatial determination threshold, determining it as an unrelated image; DTW alignment process: constructing a Euclidean matrix in the same dimension as the baseline sequence Ftext and the upgraded sequence to be aligned. Given a distance matrix D, calculate the cumulative cost matrix C (C[i][j]=D[i][j]+min(C[i−1][j],C[i][j−1],C[i−1][j−1]), C[0][0]=D[0][0], with boundary conditions of C[i][0]=C[i-1][0]+D[i][0] and C[0][j]=C[0][j-1]+D[0][j]). Backtrack from the lower right corner of C to find the optimal alignment path, adjust the sequence to be aligned, and output the feature sequence of equal length.

[0010] Preferably, in step S103, the process of the feature weight allocation submodule calculating weights based on the attention mechanism is as follows: the aligned feature sequence is mapped to the fourth preset dimension through a fully connected layer, and the weights W are output through the Softmax function. The multimodal fusion feature vector is generated by weighted summation.

[0011] Preferably, in step S104, the multimodal prompt template includes: task instructions, core text information, visual features, related information, and fusion feature summary; wherein the fusion feature summary extracts the first N principal components of Flux through principal component analysis and converts them into natural language description.

[0012] Preferably, in step S104, the constraint rules of the hierarchical relationship constraint submodule include: static rules: the font size of the first-level title is not less than a preset font size threshold and must be bolded; the second-level title must contain the core keywords of the first-level title; the number of title levels on the same page does not exceed the preset upper limit of the number of levels; dynamic logic rules: calculate the semantic similarity between the current title and the previous title; when the semantic similarity is higher than the first preset similarity threshold and the font size of the current title is smaller than the previous title, it is determined to be at the same level or a lower level; when the semantic similarity is lower than the second preset similarity threshold and the font size of the current title is larger than the previous title, it is determined to be at a higher level; when the output result of the large model does not meet the rules, re-inference is triggered. The first preset similarity threshold (denoted as θ1): is used to determine whether the semantic similarity between the current title and the previous title is too high. When the semantic similarity Sim(current title, previous title) ≥ θ1, the hierarchical rationality check is triggered. Functional Logic: If the current title's font size is smaller than the previous title (visual features indicate a possible decrease in level), but the semantic similarity is too high (e.g., "1.1 System Design" and "1.2 System Implementation" are highly semantically related), then it is determined to be at the same level or a lower level (to avoid misjudging it as a jump across levels). Second preset similarity threshold (denoted as θ2): Used to determine if the semantic similarity is "too low". When Sim(current title, previous title) ≤ θ2, level upgrade verification is triggered. Functional Logic: If the current title's font size is larger than the previous title (visual features indicate a possible increase in level), but the semantic similarity is too low (e.g., "Chapter 1 Introduction" and "2.1 Experimental Results" have a weak semantic connection), then it is determined to be at a higher level (to avoid misjudging it as at the same level due to semantic breaks).

[0013] Preferably, in step S105, the specific operations of result verification and iteration include: the multimodal joint verification engine calculates the confidence score of the result by combining the semantic progression relationship of the text and the visual style rules, and triggers manual review when the confidence score is lower than the preset confidence threshold; the model iteration triggering submodule collects error cases and marks them as training data, and for each preset number of error cases accumulated, the parameter iteration training of the optimized layout analysis module and the QWen2.5VL-72B large model enhancement module is started.

[0014] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention integrates multimodal features such as text semantics, visual layout, and image association, and combines them with DTW alignment technology to overcome the limitations of traditional OCR that relies solely on text, significantly improving the recognition accuracy of complex document title levels and enhancing adaptability to different document types.

[0015] 2. This invention ensures the logical consistency of title hierarchy by combining static rules with dynamic logic in a large model reasoning constraint; furthermore, it reduces manual review input by leveraging a closed-loop mechanism of "error case collection - model iteration", thereby achieving efficient and accurate document structure processing. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating the workflow of the multimodal feature fusion module of the present invention is shown. Figure 2 A flowchart illustrating the workflow of the cross-modal feature alignment engine of the present invention is shown; Figure 3 A flowchart of a document title hierarchy optimization method based on multimodal data processing according to the present invention is shown. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0020] Example 1: This embodiment presents a document title hierarchy optimization method based on multimodal data processing, aiming to solve the problems of high hierarchy recognition error and poor adaptability to complex documents caused by relying solely on text OCR results and ignoring multimodal features (such as document visual layout, font style, and image association information) in traditional document title hierarchy optimization. The method includes: 1. The multimodal data preprocessing module, as the core unit for data input, is responsible for collecting and processing multi-source multimodal document data, providing a high-quality data foundation for subsequent layout analysis. Internally, it includes: a PDF multimodal data parsing submodule: supporting the parsing of multimodal information such as text, images, font styles (font size, bold, color), and page layout (margins, paragraph spacing) in PDF documents; extracting text data into string format; converting image data to RGB channel PNG format (with a uniform resolution of 300 DPI); and storing font and layout information as a structured attribute dictionary. A cross-source data cleaning submodule: performing cleaning operations on the parsed multimodal data, including deleting duplicate PNG images (based on image hash value comparison, with a threshold of image hash value similarity ≥ 95%), removing blank pages (based on dual judgment of text character count and image pixel value, with a threshold of text character count < 10 and effective image pixel value < 0.01%), and repairing missing font attribute data (based on interpolation of font features of adjacent text in the same document, with a threshold of missing font attributes for three consecutive text blocks).

[0021] 2. The optimized layout analysis module is an upgrade of the original PP-DocLayout_plus-L model, incorporating multimodal feature enhancement capabilities to achieve more accurate document region classification and bounding box localization. Its core sub-modules include: Multimodal feature injection submodule: Converts the font style and page layout features output by the PDF multimodal data parsing submodule into vector form, concatenates them with the image RGB features, and inputs them into the backbone network of the PP-DocLayout_plus-L model to improve the accuracy of category prediction (especially for easily confused categories such as figure titles, table titles, and paragraph titles).

[0022] The detection box fine-tuning submodule combines text line boundary and image edge information to optimize the detection box coordinate calculation logic and perform pixel-level fine-tuning on the detection box in the title area (the detection box position offset threshold is 5 pixels, and fine-tuning is initiated when the detection box offset from the text line boundary exceeds this value; for long titles spanning two lines, the detection box height is automatically expanded to the full text line range), reducing position offset errors.

[0023] 3. The multimodal feature fusion module is responsible for fusing multimodal features from the layout analysis results, providing a more comprehensive decision-making basis for large-scale model-level recognition. It includes: a cross-modal feature alignment engine: employing the Dynamic Time Warping (DTW) algorithm to align textual semantic features (extracted based on the BERT model), visual layout features (page coordinates, font vectors), and image-related features (such as image features corresponding to the title's chart), eliminating temporal / spatial misalignment between different modalities. A feature weight allocation submodule: calculating the weights of each modal feature based on an attention mechanism (e.g., setting the weight of the title text semantic feature to 0.5, the font size feature weight to 0.3, and the page position feature weight to 0.2), and generating a multimodal fused feature vector through weighted summation.

[0024] 4. The QWen2.5VL-72B large model enhancement module optimizes the input and inference processes of the original large model, improving the logic and accuracy of hierarchical relationship recognition. Its key components include: Multimodal suggestion engineering submodule: Design a multimodal suggestion template, concatenating the initial OCR Markdown text with the multimodal fusion feature vector in a fixed format (e.g., "[Text] First-level heading candidate: XXX; [Font] Font size 24px, bold; [Position] Top 1 / 3 area of ​​the page; [Associated image]..."). Figure 1-1 This guides large models to combine multimodal information for reasoning.

[0025] Hierarchical Relationship Constraint Submodule: Sets hierarchical logic rules (such as first-level heading font size ≥ second-level heading font size, third-level headings within the same chapter must contain second-level heading keywords), performs compliance checks on the hierarchical results output by the large model, and eliminates logically contradictory results (such as a smaller font size being judged as a higher level).

[0026] 5. The Result Validation and Iteration module is responsible for evaluating the optimization results and driving model iteration, forming a closed-loop optimization mechanism. It includes: a multimodal joint validation engine: combining text semantic consistency (such as the logical progression of hierarchical headings) and visual feature consistency (such as font style rules corresponding to hierarchical levels) to calculate the confidence score. A pre-set confidence threshold of 0.8 is used; when the confidence score < 0.8, a manual review process is triggered. A model iteration triggering submodule: collects error cases found during validation (such as misjudgment of hierarchical levels, detection box offset), automatically labels them as new training data, and uses this data for parameter iteration in the optimized layout analysis module and the QWen2.5VL-72B large model enhancement module. Training is initiated every 500 accumulated error cases.

[0027] Preferably, see Figure 1 As shown, the specific configuration and working principle of the multimodal feature fusion module in this invention are as follows. The core is to solve the problem that a single text feature cannot accurately represent the title level attribute by cross-modal feature alignment and weighted fusion.

[0028] 1. Multimodal feature extraction First, multimodal feature extraction is performed on the title area data output by the optimized layout analysis module: text semantic features. The pre-trained BERT-base model is used to encode the title text, outputting a 768-dimensional vector. The calculation formula is as follows: , where T is the title text string.

[0029] Visual layout features The font size (normalized to the 0-1 range), bold indicator (0 = not bold, 1 = bold), and vertical coordinates of the page (normalized to the 0-1 range) of the title area are concatenated into a 3D vector using the following formula: ,in This is the normalized font size; 'b' indicates bold. These are normalized vertical coordinates.

[0030] Image association features ( ): If the title corresponds to an associated image (such as " Figure 1-1 Equipment structure" corresponds Figure 1-1 The ResNet50 model is used to extract image features and reduce the dimensionality to 128. If there are no associated images, zero vectors are padded, as shown in the formula: Where I represents the associated image data, It is a 128-dimensional zero vector.

[0031] 2. Cross-modal feature alignment The cross-modal feature alignment engine employs the Dynamic Time Warping (DTW) algorithm to align multimodal features across different dimensions: text semantic features... Using the baseline sequence, calculate the visual layout features. Image association features Euclidean distance matrix with reference sequence Matrix elements express The i-th element and (or The Euclidean distance to the j-th element. Find the optimal alignment path based on the distance matrix. Satisfying path continuity ( ) and completeness (covering all feature elements), after alignment, output a feature sequence of equal length. , , .

[0032] 3. Weighted Feature Fusion The feature weight allocation submodule calculates the feature weights for each modality based on an attention mechanism and then performs the fusion: Calculate attention weights: Map the aligned feature sequences to the same dimension (e.g., 256-dimensional) using a fully connected layer, and then output the weights using the Softmax function. The formula is as follows: , in .

[0033] Generating fused features: The final multimodal fused feature vector is obtained by weighted summation, as shown in the formula: This vector will serve as one of the core inputs to the QWen2.5VL-72B large model.

[0034] Preferably, see Figure 2 As shown, the specific configuration and working principle of the cross-modal feature alignment engine in this invention are as follows, which focuses on solving the fusion error problem caused by inconsistent multimodal feature dimensions and temporal / spatial misalignment.

[0035] 1. Alignment Error Assessment: First, a cross-modal feature alignment error index is defined to quantify the degree of misalignment between different modal features: Dimensional difference error ( ): Calculate the dimensionality ratio of text semantic features (768 dimensions), visual layout features (3 dimensions), and image association features (128 dimensions), using the following formula: The preset dimensional difference threshold is 10. In this case, the low-dimensional features need to be upgraded first (by mapping them to 768 dimensions through a fully connected layer).

[0036] Spatial position error : Calculate the Euclidean distance between the center coordinates of the title region and the associated image, based on their spatial location. The formula is: ,in The coordinates of the center of the title area, To correlate the image center coordinates, the spatial determination threshold is set to 1 / 2 of the page width. When the image is determined to be unrelated, it is filled with zero vectors.

[0037] 2. Dynamic Time Warping (DTW) alignment process: For feature sequences of equal length after dimensionality upgrade (e.g., all 768 dimensions), DTW alignment is performed: Constructing the distance matrix: Calculating the baseline sequence (768-dimensional) and the sequence to be aligned The Euclidean distance of each element in the (upgraded visual layout features) is used to generate a 768×768 distance matrix D, where... .

[0038] Calculate the cumulative cost matrix: Starting from the top left corner of the matrix, recursively calculate the cumulative cost at each position. The formula is: C[i][j]=D[i][j]+min(C[i−1][j],C[i][j−1],C[i−1][j−1]), where C[0][0]=D[0][0], and the boundary conditions are C[i][0]=C[i−1][0]+D[i][0] and C[0][j]=C[0][j−1]+D[0][j].

[0039] Backtracking to find the optimal path: Starting from the bottom right corner of the cumulative cost matrix (C

[767]

[767] ), backtrack to find the source of minimum cost at each step (top, left, top left) to obtain the optimal alignment path P. Adjust the element order of the sequence to be aligned according to the path, and output the sequence aligned with the reference sequence. .

[0040] 3. Alignment result optimization: An alignment error correction mechanism is introduced to further improve alignment accuracy. Calculate the mean square error (MSE) of the aligned sequence: The mean squared error (MSE) threshold for the alignment result is 0.1. When MSE > 0.1, the distance calculation weights of DTW are readjusted (e.g., the weights of the feature dimensions corresponding to the text keywords are increased).

[0041] Add spatial constraints: If the title area is located at the top of the page, when aligning, prioritize aligning the "vertical coordinate" dimension in the visual layout features with the "hierarchical keywords" (such as Chapter 1, 1.1) dimension in the text semantic features to ensure consistency between spatial position and semantic hierarchy.

[0042] Preferably, the specific configuration and working principle of the QWen2.5VL-72B large model enhancement module in this invention are as follows: through multimodal prompts and logical constraints, the ability of the large model to recognize the hierarchical relationship of the title is improved.

[0043] 1. Multimodal Prompt Template Design: The multimodal prompt engineering submodule designs structured prompt templates to transform multimodal information into an input format understandable by the larger model. An example template is shown below: Recognize document heading levels and output heading levels 1 (#), 2 (##), 3 (###), 4 (####), and 5 (#####). Modify only the Markdown levels without changing the main text content.

[0044] Text content: {T} (T is the title text); Visual features: Font size = {s}px, bold = {b}, page position = {p} (top / middle / bottom); Association information: Whether to associate image = {flag}, Associated image description = {desc} (if any); Fusion Feature Summary: { }( (Top 5 key dimensions of multimodal fusion features) [Output Requirements] Output only the title text with Markdown hierarchy, no additional content.

[0045] in,{ } Feature vectors fused from multimodal modes The top 5 principal components were extracted using principal component analysis (PCA) and converted into natural language descriptions (e.g., principal component 1: 35% of the semantic features are related to the chapter).

[0046] This prompt template will serve as the core input format for the QWen2.5VL-72B large model.

[0047] 2. Hierarchical Relationship Reasoning Constraints: The hierarchical relationship constraint submodule sets two types of constraint rules to guide and verify the reasoning process of the large model: Static rule constraints: Based on the document editing specifications, fixed rules are preset, including first-level headings with a font size of ≥20px and must be bolded, second-level headings must contain the core keywords of the first-level headings (e.g., if the first-level heading is 1 System Design, the second-level headings must contain related vocabulary for System Design), the number of levels within the same page is ≤5, and the output results of the large model must meet all static rules, otherwise re-inference is triggered.

[0048] Dynamic logical constraints: Based on the semantic connections and visual feature changes of the context headings, the rationality of the hierarchy is dynamically determined. For example, the semantic similarity between the current heading and the previous heading is calculated (based on cosine similarity). Dynamic logical constraints: Based on the semantic association and visual feature changes of the context title, the rationality of the hierarchy is dynamically judged. Specific rules include: Case 1: If the semantic similarity Sim between the current title and the previous title is higher than the preset upper limit threshold (e.g., Sim>0.6), and the font size of the current title is smaller than that of the previous title, then the current title is determined to be at the same or lower level as the previous title.

[0049] Scenario 2: If the semantic similarity Sim between the current title and the previous title is lower than the preset lower threshold (e.g., Sim < 0.3), and the font size of the current title is larger than that of the previous title, then the current title is determined to be at a higher level than the previous title.

[0050] When the hierarchical results output by the large model do not meet the above dynamic logic rules, re-inference will be triggered.

[0051] 3. Result Optimization and Output: Final optimization is performed on the hierarchical results output by the large model: Hierarchical conflict correction: When there is a hierarchical logical conflict between adjacent titles (such as "##1.1 Functional Modules" followed by "#1.2 Performance Indicators"), the incorrect hierarchy will be adjusted to "###1.2 Performance Indicators" based on visual features (font size, position) and semantic similarity.

[0052] Standardize the format: unify the spacing between Markdown hierarchy symbols and heading text (e.g., "#Heading 1" instead of "#Heading 1"), remove unnecessary spaces and special characters, and ensure that the output format conforms to common document specifications.

[0053] The beneficial effects of this embodiment are as follows: by using multimodal feature fusion and DTW alignment technology, the accuracy of identifying the title hierarchy of complex documents is improved; by combining large model reasoning with static / dynamic constraint rules, the logic is enhanced; and the closed-loop optimization mechanism continuously iterates the model, reducing the cost of manual review and achieving efficient and accurate document structuring processing.

[0054] Example 2: See Figure 3 As shown, the present invention also provides a document title hierarchy optimization method based on multimodal data processing, the method comprising the following steps: Step S101: Multimodal Document Data Acquisition and Preprocessing. Using the PDF multimodal data parsing submodule of the multimodal data preprocessing module, various types of PDF documents (textbooks, academic papers, company annual reports, etc.) are acquired, and multimodal information such as text, images, font styles, and page layout is parsed. Using the cross-source data cleaning submodule, image resolution is unified to 300 DPI, duplicate PNG images (duplicate image determination threshold: image hash value similarity ≥ 95%) and blank pages (blank page determination threshold: text character count < 10 and effective image pixel value < 0.01%) are removed, and missing font attribute data is repaired (font attribute missing determination threshold: 3 consecutive text blocks). The cleaned multimodal dataset is then output.

[0055] Step S102: Multimodal Enhanced Layout Analysis. The preprocessed multimodal data is input into the optimized layout analysis module. Through the multimodal feature injection sub-module, font, layout features, and image features are fused and input into the model backbone. The detection box fine-tuning sub-module (with a detection box position offset threshold of 5 pixels) optimizes the coordinates of the title region detection boxes, outputting layout analysis results containing 21 categories of detection boxes (with newly added independent categories for figure titles and table titles). Simultaneously, multimodal features of each title region are extracted.

[0056] Step S103: Multimodal Feature Alignment and Fusion. In the multimodal feature fusion module, the DTW algorithm is used to align the textual semantic features, visual layout features, and image association features of the title through a cross-modal feature alignment engine (the spatial judgment threshold is set to 1 / 2 of the page width, and the preset dimensional difference threshold is 10), outputting a feature sequence of equal length. Based on the attention mechanism of the feature weight allocation submodule, the feature weights of each modality are calculated, and a weighted sum is generated to produce a multimodal fusion feature vector. After alignment, the mean squared error (MSE) is calculated. The threshold for the mean squared error (MSE) of the alignment result is 0.1. When MSE > 0.1, the DTW distance calculation weights are readjusted.

[0057] Step S104: Large-scale multimodal reasoning and hierarchical recognition construct a multimodal prompt template, combining the initial OCR Markdown text, multimodal feature description, The abstract is concatenated as prompt input and passed to the QWen2.5VL-72B large model. The static rules and dynamic logical constraints of the hierarchical relationship constraint submodule are enabled (the semantic similarity threshold is set to Sim>0.6 and Sim<0.3) to guide the large model to infer the title hierarchy and output the preliminary Markdown format hierarchy results.

[0058] Step S105: Result Verification and Model Iteration. Using the multimodal joint verification engine of the result verification and iteration module, combined with semantic consistency and visual feature consistency, the result confidence score is calculated (preset confidence threshold is 0.8). Manual review is triggered when the confidence score is <0.8. Error cases discovered during manual review are collected. Through model iteration, a sub-module is triggered to label these error cases as training data. Every 500 accumulated cases, parameter iteration training of the optimized layout analysis module and the large model enhancement module is initiated, forming a closed-loop optimization.

[0059] The beneficial effects of this embodiment are as follows: by using multimodal feature fusion and DTW alignment technology, the accuracy of identifying the title hierarchy of complex documents is improved; by combining large model reasoning with static / dynamic constraint rules, the logic is enhanced; and the closed-loop optimization mechanism continuously iterates the model, reducing the cost of manual review and achieving efficient and accurate document structuring processing.

[0060] All formulas in this invention are dimensionless and calculated numerically. The preset parameters in the formulas can be set by those skilled in the art according to the actual situation.

[0061] The weighting coefficients of this invention are used to measure the degree of influence of different factors or variables on a certain outcome or decision. The weighting coefficient is defined as the numerical value assigned to each factor when comparing and evaluating multiple factors, reflecting their importance or priority. These weighting coefficients can be determined according to specific circumstances and needs, and are usually jointly formulated and confirmed by professionals or relevant stakeholders. By reasonably setting the weighting coefficients, programs or systems can be helped to make decisions or predictions more accurately.

[0062] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0063] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A document title hierarchy optimization method based on multimodal data processing, characterized in that, Includes the following steps: Step S101: Multimodal document data acquisition and preprocessing; Step S102: Input the preprocessed multimodal data into the optimized layout analysis module, convert the font style and page layout features into vectors and concatenate them with the image RGB features, then input them into the model backbone, optimize the coordinates of the title region detection box, and output the layout analysis results of multiple detection boxes and the multimodal features of each title region; Step S103: Use the dynamic time warping algorithm to align the text semantic features, visual layout features and image association features of the title, output an equal-length feature sequence, calculate the weight of each modality feature, and generate a multimodal fusion feature vector by weighted summation; Step S104: Concatenate the initial OCR Markdown text, multimodal feature description, and multimodal fusion feature vector summary into a prompt input and feed it into the QWen2.5VL-72B large model. Enable static rules and dynamic logical constraints to guide the large model's inference and output the preliminary results of the title level in Markdown format. Step S105: Calculate the confidence score of the results by combining text semantic consistency and visual feature consistency through a multimodal joint verification engine. When the confidence score is lower than the preset confidence threshold, trigger manual review, collect error cases found during the review, mark the error cases as training data, and drive the parameter iterative training of the optimized layout analysis module and the QWen2.5VL-72B large model enhancement module.

2. The document title hierarchy optimization method based on multimodal data processing according to claim 1, characterized in that, In step S101, specifically: the text data is converted into a string format, the image data is converted into RGB channel PNG format with a unified resolution of the preset resolution, and the font style and page layout information are stored as a structured attribute dictionary; duplicate PNG images are deleted based on image hash value comparison, blank pages are removed by dual determination of text character count and image pixel value, and missing font attribute data is repaired by interpolation based on the font features of adjacent text in the same document.

3. The document title hierarchy optimization method based on multimodal data processing according to claim 1, characterized in that, In step S102, the multimodal feature injection submodule converts font style and page layout features into vectors, and then concatenates them with image RGB features to input the backbone network of the model; the detection box fine adjustment submodule combines text line boundary and image edge information to automatically expand the height of long title detection boxes spanning multiple lines to the full text line range.

4. The document title hierarchy optimization method based on multimodal data processing according to claim 1, characterized in that, In step S103, a multimodal feature extraction step is included before the multimodal feature alignment and fusion, specifically: Text semantic features: The title text is encoded using a pre-trained BERT-base model, and the output is a vector with the first preset dimension; Visual layout features: Normalize the font size of the title area to the 0-1 range, bold the font, normalize the vertical coordinates of the page to the 0-1 range, and concatenate them into a vector with a second preset dimension; Image association features: If the title corresponds to an associated image, the ResNet50 model is used to extract image features and reduce the dimensionality to the third preset dimension; if there is no associated image, the zero vector of the third preset dimension is filled.

5. The document title hierarchy optimization method based on multimodal data processing according to claim 1, characterized in that, In step S103, the process of employing the dynamic time warping algorithm includes: Alignment error evaluation: Calculate dimensionality difference error and spatial location error. When the dimensionality difference error exceeds the preset dimensionality difference threshold, the low-dimensional features are mapped to the target dimension consistent with the semantic feature dimension of the text through a fully connected layer. When the spatial location error exceeds the preset spatial determination threshold, it is determined to be an unrelated image. DTW alignment process: Construct a Euclidean distance matrix D in the same dimension square form between the baseline sequence Ftext and the sequence to be aligned after dimensionality increase. Calculate the cumulative cost matrix C, C[i][j]=D[i][j]+min(C[i−1][j],C[i][j−1],C[i−1][j−1]), C[0][0]=D[0][0]. The boundary conditions are C[i][0]=C[i-1][0]+D[i][0] and C[0][j]=C[0][j-1]+D[0][j]. Backtrack from the lower right corner of C to find the optimal alignment path, adjust the sequence to be aligned, and output a feature sequence of equal length.

6. The document title hierarchy optimization method based on multimodal data processing according to claim 1, characterized in that, In step S103, the process of calculating weights based on the attention mechanism by the feature weight allocation submodule is as follows: the aligned feature sequence is mapped to the fourth preset dimension through a fully connected layer, and the weight W is output through the Softmax function. The multimodal fusion feature vector is generated by weighted summation.

7. The document title hierarchy optimization method based on multimodal data processing according to claim 1, characterized in that, In step S104, the multimodal prompt template includes: task instructions, core text information, visual features, related information, and fusion feature summary; wherein the fusion feature summary extracts the first N principal components of Flux through principal component analysis and converts them into natural language description.

8. The document title hierarchy optimization method based on multimodal data processing according to claim 1, characterized in that, In step S104, the constraint rules of the hierarchical relationship constraint submodule include: static rules: the font size of the first-level title is not less than a preset font size threshold and must be bolded; the second-level title must contain the core keywords of the first-level title; the number of title levels on the same page does not exceed the preset upper limit of the number of levels; dynamic logic rules: calculate the semantic similarity between the current title and the previous title; when the semantic similarity is higher than the first preset similarity threshold and the font size of the current title is smaller than the previous title, it is determined to be at the same level or a lower level; when the semantic similarity is lower than the second preset similarity threshold and the font size of the current title is larger than the previous title, it is determined to be at a higher level; when the output result of the large model does not meet the rules, re-inference is triggered.

9. The document title hierarchy optimization method based on multimodal data processing according to claim 1, characterized in that, In step S105, the specific operations of result verification and iteration include: the multimodal joint verification engine calculates the confidence score of the result by combining the semantic progression relationship of the text and the visual style rules, and triggers manual review when the confidence score is lower than the preset confidence threshold; the model iteration triggering submodule collects error cases and marks them as training data, and for each preset number of error cases accumulated, the parameter iteration training of the optimized layout analysis module and the QWen2.5VL-72B large model enhancement module is started.

Images