Multi-modal document content cross-platform analysis system

Abstract

The invention provides a multi-modal document content cross-platform analysis system, and relates to the technical field of data processing, and the system comprises a document preprocessing module which is used for receiving multi-source heterogeneous document input data, and generating a preprocessed document through format conversion, page segmentation, noise reduction and optical character recognition; the feature extraction module is used for extracting three types of features, including spatial layout features, semantic features and logic structure features, based on the preprocessed document; according to the method, the multi-source heterogeneous document is cooperatively processed, accurate extraction, calibration and association of features in cross-platform analysis are realized, finally structured data are generated, and the accuracy, consistency and cross-platform applicability of multi-modal document analysis are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal document content cross-platform parsing system. Background Art

[0002] In cross-platform document processing scenarios, such as automatically extracting key terms and structural information from scanned contract files, users often come into contact with documents from a variety of sources, such as PDF, Word, and scanned documents. These documents contain both text semantics, such as the specific content of the terms, and important spatial layout information; for example, the location of signatures, table borders, and logical structures, such as the hierarchy of chapter titles and list items of terms. These are critical requirements for downstream tasks such as intelligent review and data warehousing.

[0003] However, existing technologies often have limitations when processing documents with complex layouts or physical distortions (such as scanning tilt and image noise): they may not be efficient enough in the coordinated use of text semantics, precise spatial position and logical structure information, which may lead to deviations in the hierarchical relationships and semantic type recognition of complex elements (such as nested tables and multi-level lists). Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a cross-platform parsing system for multimodal document content to achieve effective integration of text semantics, precisely calibrated spatial layout features and logical structure information.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows: First, a multimodal document content cross-platform parsing system, including: The document preprocessing module is used to receive multi-source heterogeneous document input data and generate preprocessed documents through format conversion, page segmentation, noise reduction and optical character recognition; Feature extraction module, which is used to extract three types of features based on preprocessed documents, including spatial layout features, semantic features, and logical structure features; The feature calibration module is used to use spatial layout features to construct a document spatial reference system based on the statistical values ​​of text line tilt angles and the coordinates of the table frame corners. The module selects the connection points of text lines, the physical boundary corners of tables, and the center points of paragraph spacing as spatial reference points, and generates spatial calibration parameters based on the coordinate offsets between the reference points. The module then outputs the calibrated spatial layout features. The feature association module is used to collaboratively analyze semantic features, logical structure features, and calibrated spatial layout features. It generates multimodal association features by mapping and verifying semantic content and spatial location, and matching and associating logical structure with spatial distribution. The type recognition module is used to input multimodal correlation features into a pre-trained visual-linguistic multimodal large model, and output the parent-child hierarchical relationship and semantic type identification of document elements through combined semantic understanding and structural analysis; The data generation module is used to construct a tree-like semantic structure based on the parent-child hierarchical relationship and semantic type identification, with chapters as parent units, paragraphs as child units, and nested tables and list items as terminal units, and convert the tree-like semantic structure into structured data output.

[0006] In a second aspect, a computing device includes: one or more processors; The storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the system.

[0007] According to a third aspect, a computer-readable storage medium stores a program, which implements the system when executed by a processor.

[0008] The above solution of the present invention includes at least the following beneficial effects: Documents support multi-source input, including PDF, Word, and scanned documents. Combined with operations like format conversion and noise reduction, it can effectively handle documents with physical distortion (such as scanning tilt and image noise), solving the adaptation challenges of cross-format and cross-platform document processing. It captures three types of features: spatial layout, semantics, and logical structure. Combined with the spatial reference point calibration mechanism of the feature calibration module, it eliminates the impact of layout offset on feature accuracy. By mapping semantics to spatial positions and matching logical structures to spatial distributions, it overcomes the limitations of single-modal information processing, and is particularly accurate in identifying relationships between complex elements such as nested tables and multi-level lists. Joint analysis based on a large visual-linguistic multimodal model accurately outputs parent-child hierarchical relationships and semantic types (such as chapters, tables, and list items) of elements. By constructing a tree-like semantic structure, it provides standardized, usable structured data for downstream tasks (such as intelligent review and data warehousing). BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 Schematic diagram of a multimodal document content cross-platform parsing system provided by an embodiment of the present invention.

[0010] Figure 2 This is a flow chart of receiving multi-source heterogeneous document input data and generating pre-processed documents through format conversion, page segmentation, noise reduction and optical character recognition, provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0012] like Figure 1 As shown, an embodiment of the present invention proposes a multimodal document content cross-platform parsing system, including: The document preprocessing module is used to receive multi-source heterogeneous document input data and generate preprocessed documents through format conversion, page segmentation, noise reduction and optical character recognition; Feature extraction module, which is used to extract three types of features based on preprocessed documents, including spatial layout features, semantic features, and logical structure features; The feature calibration module is used to use spatial layout features to construct a document spatial reference system based on the statistical values ​​of text line tilt angles and the coordinates of the table frame corners. The module selects the connection points of text lines, the physical boundary corners of tables, and the center points of paragraph spacing as spatial reference points, and generates spatial calibration parameters based on the coordinate offsets between the reference points. The module then outputs the calibrated spatial layout features. The feature association module is used to collaboratively analyze semantic features, logical structure features, and calibrated spatial layout features. It generates multimodal association features by mapping and verifying semantic content and spatial location, and matching and associating logical structure with spatial distribution. The type recognition module is used to input multimodal correlation features into a pre-trained visual-linguistic multimodal large model, and output the parent-child hierarchical relationship and semantic type identification of document elements through combined semantic understanding and structural analysis; The data generation module is used to construct a tree-like semantic structure based on the parent-child hierarchical relationship and semantic type identification, with chapters as parent units, paragraphs as child units, and nested tables and list items as terminal units, and convert the tree-like semantic structure into structured data output.

[0013] In an embodiment of the present invention, the document supports multi-source input such as PDF, Word, and scanned documents. Combined with operations such as format conversion and noise reduction, it can effectively deal with documents with physical distortion (such as scanning tilt and image noise), solving the adaptation problem of cross-format and cross-platform document processing; it captures three types of features: spatial layout, semantics, and logical structure, and combines the spatial reference point calibration mechanism of the feature calibration module to eliminate the impact of layout offset on feature accuracy; through the mapping verification of semantics and spatial position, and the matching association of logical structure and spatial distribution, it breaks the limitations of single-modal information processing, especially for the more accurate identification of association relationships of complex elements such as nested tables and multi-level lists. Based on the joint analysis of the visual-language multimodal large model, the parent-child hierarchical relationship and semantic type (such as chapters, tables, list items) of elements can be accurately output, and then a tree-shaped semantic structure can be constructed through the data generation module to provide standardized and usable structured data for downstream tasks (such as intelligent review and data warehousing).

[0014] like Figure 2 As shown, in another preferred embodiment of the present invention, receiving multi-source heterogeneous document input data and generating a pre-processed document through format conversion, page segmentation, noise reduction and optical character recognition may include: Receive multi-source heterogeneous document input data, perform format conversion operations to generate converted documents in a unified image format; Perform page separation on the converted document and output independent page images; Perform interference information elimination operations on independent page images and output optimized page images; Optimize page images through optical character recognition processing and output recognition data containing text content and character position coordinates; A pre-processed document is generated based on the recognition data, and the pre-processed document includes text content, character position set, and distribution information of text and non-text areas within the page.

[0015] In an embodiment of the present invention, multi-source document reception and format recognition: the system receives documents through interfaces compatible with multiple channels such as HTTP, FTP, and local file directories, and simultaneously initiates format detection to parse the document header metadata (such as file suffix, MIME type identifier), and quickly identify the document format (such as the "%PDF-" identifier of a PDF file, the "PK" compressed package identifier of a Word document); for documents with fuzzy formats (such as files without suffixes), the type is further confirmed through content feature matching (such as the cross-reference table structure of PDF, the OOXML tag of Word) to ensure that no omissions are missed.

[0016] PDF document conversion: Text-based PDF: By parsing the internal object stream, the text blocks, font information, and coordinate data of each page are extracted. The vector rendering engine is then called to redraw the text as an image according to the original layout, ensuring that the text size and spacing are consistent with the original document.

[0017] Scanned PDF: Reads image objects (such as the Image type in XObject) page by page, extracts pixel matrix data, and directly converts it into an image with the same resolution as the original (such as 300DPI TIFF), preserving the original visual details of the scanned document.

[0018] Office document conversion: Word / Excel / PPT: Call the document parsing engine to parse the XML structure (such as Word's document.xml and PPT's slide.xml), extract layout information such as paragraph style (font, size, color), table borders, and chart coordinates, and then render it into images by page through virtual printing technology to ensure that the continuity of table lines and the layout of formulas are exactly the same as the original document.

[0019] Image to web page conversion: Scanned images (JPG / PNG): Directly read pixel data. If the resolution is lower than 200 DPI, automatic interpolation and amplification are performed to ensure clarity for subsequent OCR recognition. If there is color deviation (such as a yellowish background), white balance correction is performed to unify the color tone.

[0020] HTML web pages: Load web pages through the browser's core rendering engine, parse CSS layout rules and DOM structure, convert dynamic content (such as tables generated by JavaScript) into static images, and block irrelevant elements such as advertisements and floating windows, retaining only the core document content.

[0021] All converted images are uniformly formatted in 24-bit true color TIFF (supporting multi-page storage) and named according to the document source and page number (such as "PDF_Contract1_Page1.tif"). At the same time, information such as the original document format and conversion time are embedded in the image metadata.

[0022] Page boundary detection: For continuous multi-page images (such as long scanned images), a 50×50 pixel detection window is vertically slid in steps of 10 pixels. The average brightness value within the window is calculated (0-255). When the average brightness value of three consecutive windows exceeds 240 (close to pure white, indicating a page gap), the starting row coordinates of the area are recorded. For multi-page images converted from PDF, the original PDF page size (e.g., A4 paper is 210mm wide and 297mm high) is converted to pixel dimensions (2480×3508 pixels at 300 DPI). The detected gaps are verified to conform to the rule of "page height = 3508 pixels ± 50 pixels." If the deviation exceeds 100 pixels, the detection window size is resized (e.g., to 100×100 pixels).

[0023] Single page coordinate division: The upper boundary of the first page is the top of the image (pixel row 0), the lower boundary is the first detected gap start row (such as pixel row 3508), the left boundary is the leftmost side of the image (pixel column 0), and the right boundary is the rightmost side of the image (pixel column 2479), forming a rectangular range (0, 0)-(2479, 3508).

[0024] The upper boundary of the second page is the next line below the lower boundary of the first page (the 3509th pixel line), and the lower boundary is the second gap starting line (such as the 7017th pixel line), and so on. For tilted scans, first detect the edge angle through Hough transform (such as 3° tilted to the left), rotate 3° to the right around the center point of the image, and then divide the pages according to the above method to prevent the separated pages from containing the content of adjacent pages.

[0025] Page Number Marking and Verification: For separated single-page images, OCR is used to identify the page number text in the footer (e.g., "Page 3 of 5") and mark the pages in numerical order. If no page number is recognized, it is marked in the order of separation (e.g., "001," "002"). The aspect ratio of each page is calculated (e.g., 2480 / 3508 ≈ 0.707, which matches the aspect ratio of A4 paper). If the aspect ratio of a page deviates by more than 5% (e.g., 0.65 or 0.78), the page gap is rechecked to avoid page merging or splitting errors caused by missed detections.

[0026] Noise detection and cancellation: Salt and pepper noise: traverse each pixel of the image and check the other 8 pixels within a 3×3 pixel range; if the central pixel is pure black (0) and the surrounding 8 pixels are pure white (255), or the center is pure white and the surrounding pixels are pure black, it is judged to be salt and pepper noise; when eliminating it, replace the central pixel with the median of the surrounding 8 pixels (if the surrounding pixels are mostly pure white, change the black noise point to pure white).

[0027] Gaussian noise: Calculate the standard deviation of the pixel values ​​of each 10×10 pixel block. If the standard deviation is less than 15 (the pixel values ​​are too evenly distributed and appear blurred), apply a Gaussian filter to the block: the new value of each pixel is the weighted average of the pixels within the surrounding 5×5 range (the closer the distance, the higher the weight. The specific weight distribution is: 0.25 for the center pixel; 0.15 for each of the four pixels adjacent to the center pixel in the horizontal or vertical direction (i.e., the pixels immediately above, below, to the left, and to the right of the center pixel); 0.15 for each of the four pixels adjacent to the center pixel in the diagonal direction (i.e., the pixels immediately above, below, to the left, and to the right of the center pixel); The 4 pixels with a horizontal or vertical interval of 1 pixel from the center pixel (i.e., the 2nd pixel above, 2nd pixel below, 2nd pixel to the left, and 2nd pixel to the right of the center pixel) are 0.05 each; and the 4 pixels with a diagonal interval of 1 pixel from the center pixel (i.e., the 2nd pixel to the upper left, 2nd pixel to the upper right, 2nd pixel to the lower left, and 2nd pixel to the lower right of the center pixel) are 0.02 each) are used to sharpen the blurred area while preserving the sharpness of the text edges (e.g., the horizontal and vertical strokes of the letter "T" are clear at their junction).

[0028] Edge shadows: Use edge detection algorithms (such as the Sobel operator) to identify the effective content boundary of the page and calculate the extent of the dark area (pixel brightness less than 100) outside the boundary (left dark area width = x-coordinate value of the effective content left boundary minus x-coordinate value of the image left edge (x-coordinate value of the image left edge is 0); right dark area width = x-coordinate value of the image right edge minus x-coordinate value of the effective content right boundary; upper dark area width = y-coordinate value of the effective content upper boundary minus y-coordinate value of the image upper edge (y-coordinate value of the image upper edge is 0); lower dark area width = y-coordinate value of the image lower edge minus y-coordinate value of the effective content lower boundary). If the dark area is at the edge of the page and is wider than 5 mm (approximately 58 pixels at 300 DPI), the dark area is cropped (for example, if the left dark area is 100 pixels wide, adjust the left edge of the page from x=0 to the x-coordinate value of the effective content left boundary) to ensure that only the effective content is retained after cropping.

[0029] Tilt correction: Ten text-dense areas (such as the middle of a paragraph) are randomly selected from the image. For each area, the Hough transform is used to detect the baseline angle of the text lines. For example, the detected tilt angles of the ten lines of text are -2.8°, -3.1°, and -2.9° (a negative sign indicates a leftward tilt). The average of -3° is taken as the overall tilt angle of the page. Correction is performed by rotating the image 3° to the right, using the image center as the axis. This involves adjusting the coordinates to determine the new (x, y) position of each pixel, thereby horizontalizing the originally left-tilted text lines. After correction, the blank areas at the edges of the image are filled with pure white to avoid black edges.

[0030] Contrast Enhancement: Calculate the brightness histogram of the image, find the brightness values corresponding to 5% (low threshold, e.g., 60) and 95% (high threshold, e.g., 200) of the cumulative pixel ratio. Adjust the pixels below the low threshold to 0 (pure black), those above the high threshold to 255 (pure white), and stretch the intermediate pixels proportionally (e.g., a pixel with an original brightness of 100 becomes (100 - 60) / (200 - 60)×255), making the difference between the text and the background more significant.

[0031] Text area localization: Convert the image to grayscale, calculate the average brightness for each 20×20 pixel block, and mark the blocks with an average brightness less than 127 as potential text blocks; merge adjacent text blocks (spacing < 10 pixels) to form text areas, and remove non-text areas with an aspect ratio > 10 or < 0.1 (such as thin table lines).

[0032] Text line segmentation: Vertically scan within the text area. If the ratio of black pixels in two consecutive lines is > 50%, they belong to the same line; if the line spacing is > 1.5 times the line height (e.g., if the line height is 20 pixels, the spacing > 30 pixels), they are divided into different lines.

[0033] Single character segmentation: Horizontally scan the text line and detect the blank between characters (more than 3 consecutive 255 pixels). Chinese characters are segmented based on an average width of 20 pixels, and English words are segmented based on the letter spacing (e.g., the spacing between "a" and "b" is 5 pixels), ensuring that "合同" is split into "合" and "同", and "apple" is split into "a", "p", "p", "l", "e".

[0034] Extract the character contour features (such as the outer frame and internal structure of the character "合"), compare them with the character library templates, and confirm the character if the matching degree > 9�% (e.g., if the matching degree with the "合" character template is 95%, it is recognized as "合"). Taking the upper left corner of the page as the origin, mark the upper left corner (x1, y1) and the lower right corner (x2, y2) of the character's bounding rectangle (e.g., the coordinates of the "合" character are (150, 300, 180,

[0035]

[0036] 330)), and simultaneously mark the recognition confidence (e.g., 98%); organize them hierarchically according to "page → text area → text line → character", and output the complete text (e.g., "第一条甲方义务"), the character coordinate set (stored by line), and the low confidence marks (characters with < 60% confidence are marked as "[to be confirmed]").

[0035] Text content recombination: Stitch the characters together in ascending order of the y coordinate of the text line and from left to right in the x coordinate to form paragraphs; when the line spacing > 2 times the line height (e.g., if the line height is 20 pixels, the spacing > 40 pixels), it is divided into new paragraphs. For example, if the distance between "第一条" and "1.1内容" is 50 pixels, they are divided into two paragraphs. The table content is sorted by row and column according to the cell coordinates (determined by detecting the table lines). For example, "序号" in cell A1 and "名称" in cell B1 form the first row.

[0036] Text line coordinates: Calculate the baseline of each line (the y - coordinate corresponding to the average of the y - coordinates of the bottom positions of all characters in the vertical direction for the bottom edge of the characters in that line, e.g., for the line "Zhang San", the baseline y = 320), the line width (from the x1 of the first character to the x2 of the last character, e.g., 80 pixels), and record the line coordinates (e.g., (100, 300, 180, 330)).

[0037] Paragraph coordinates: Calculate the minimum bounding rectangle of the paragraph (from x1, y1 of the first line to x2, y2 of the last line), e.g., for a 3 - line text paragraph, the coordinates are (90, 290, 200, 400).

[0038] Analysis of non - text areas: Image area: Mark the areas that are not covered by text and have rich pixel colors (RGB value difference > 50), and record the coordinates (e.g., (300, 200, 500, 400)) and dimensions; Table border: Identify horizontal / vertical lines (pixels are 0, length > 50 pixels), record the start and end points (e.g., for a horizontal line (100, 200)-(500, 200)), and associate the enclosed text area; Blank area: Mark the areas without text, images, or tables (e.g., footer blank (0, 3400, 2480, 3508)).

[0039] The pre - processed document contains three layers of information: Text layer: The text divided by paragraphs and the confidence level (e.g., "Paragraph 1: 96%"); Location layer: Pixel - level coordinates of characters, lines, and paragraphs; Area layer: The type, coordinates, and attributes of non - text areas (e.g., a table with 5 rows and 3 columns).

[0040] Through unified format conversion, the format differences of multi - source heterogeneous documents (such as PDF, Word, scanned documents) are solved, ensuring that the system can process various common document types; page segmentation and noise reduction operations eliminate problems such as multi - page adhesion, noise interference, and skew distortion. During the OCR process, the character position coordinates are recorded synchronously, providing accurate spatial data support for subsequent analysis of the document layout (such as paragraph spacing, table structure), and meeting the parsing requirements for the spatial characteristics of the document. The pre - processed document integrates text content, location information, and area distribution, providing comprehensive and standardized data input for the feature extraction module (such as spatial layout features, semantic features).

[0041] In a preferred embodiment of the present invention, based on the pre - processed document, three types of features are extracted, including spatial layout features, semantic features, and logical structure features, which may include: Based on the set of character positions and the distribution information of text and non - text regions in the pre - processed document, extract spatial layout features, where the spatial layout features include text line boundary coordinates, character spacing data, table frame corner coordinates, and non - text region positions; based on the text content in the pre - processed document, extract semantic features, where the semantic features include entity object identifiers, syntactic structure relationships, and semantic association relationships; based on the paragraph segmentation identifiers, title font size differences, list symbol sequences, and table frame topological relationships in the pre - processed document, extract logical structure features, where the logical structure features include chapter level numbers, paragraph attribution relationships, list nesting levels, and table cell logical structures.

[0042] In an embodiment of the present invention, based on the set of character positions in the pre - processed document, group and count character coordinates by text line: For each line of text, traverse the upper - left corner (x1, y1) and lower - right corner (x2, y2) coordinates of all characters, take the minimum value of x1 of all characters in the line as the left boundary of the text line, and the maximum value of x2 as the right boundary; Take the minimum value of y1 of all characters in the line as the upper boundary, and the maximum value of y2 as the lower boundary, to form the rectangular boundary of the text line (e.g., "left = 100" means the pixel number of the left edge of the text line from the starting point of the upper - left corner of the document image in the horizontal direction is 100, "right = 500" means the pixel number of the right edge of the text line from the starting point in the horizontal direction is 500, "up = 200" means the pixel number of the upper edge of the text line from the starting point in the vertical direction is 200, "down = 220" means the pixel number of the lower edge of the text line from the starting point in the vertical direction is 220); For long texts that span multiple lines (such as paragraphs with automatic line breaks), split them according to the actual line - break positions to ensure that each text line boundary only contains single - line characters.

[0043] For characters within the same text line, calculate the horizontal and vertical spacing between adjacent characters: Horizontal spacing: the x1 of the next character minus the x2 of the previous character (e.g., if x2 of the character "Zhang" = 150 and x1 of the character "San" = 155, then the spacing = 5 pixels); Vertical spacing: count the y1 differences of all characters in the same line, and take the average value as the vertical deviation within the line (e.g., if most characters have y1 = 200 and a few have 202, then the average deviation = 1 pixel); Summarize the spacing data by text line to form features such as "average horizontal spacing", "maximum spacing", "minimum spacing", etc. (e.g., for a certain line, the average horizontal spacing = 3 pixels, the maximum spacing = 10 pixels, corresponding to the space after punctuation).

[0044] Combined with the non - text region distribution information in the pre - processed document, locate the corner points of the table border: Identify the outer frame lines of the table (horizontal and vertical lines, determined by the "table border" marker in the non-text area); find the intersection points of the horizontal lines and vertical lines: the intersection of the upper border and the left border is the upper left corner point (x1, y1), the intersection of the upper border and the right border is the upper right corner point (x2, y1), the intersection of the lower border and the left border is the lower left corner point (x1, y2), and the intersection of the lower border and the right border is the lower right corner point (x2, y2); for nested tables (tables containing sub-tables inside), extract the corner points in the order from the outside to the inside, such as first recording the four corner points of the outer table, and then recording the four corner points of the inner table, and distinguishing them with hierarchical markers (such as "outer upper left", "inner upper left").

[0045] Based on the text and non-text area distribution information in the preprocessed document, determine the boundaries of the non-text areas: Image area: Take the upper left corner (x1, y1) and the lower right corner (x2, y2) of the outer rectangle of the image. For example, if the picture is on the right side of the page, the coordinates are (600, 300, 800, 500); Blank area: Count the continuous areas on the page that do not contain text and images, and take the coordinates of their minimum enclosing rectangle. For example, the coordinates of the footer blank area are (0, 3400, 2480, 3508); For each non-text area, mark the type (image / blank / table outer frame, etc.) and the relative position to the surrounding text (such as "the image is below the paragraph, spacing = 50 pixels").

[0046] Based on the text content of the preprocessed document, locate entity objects through rule matching and feature recognition: Personal names: Match the surname library (such as "Zhang, Li, Wang") + name features (2-3 Chinese characters, without obvious function words), combined with appellations (such as "Mr., Ms."), such as "Mr. XX" is marked as "personal name"; Place names: Match the administrative division names (such as "City AA, District BB") + geographical feature words (such as "street, road"), such as "Jianguo Road, District BB, City AA" is marked as "place name"; Dates: Identify formats such as "year / month / day", "YYYY-MM-DD", etc. For example, "October 1, 2023" and "2023-10-01" are marked as "dates"; Amounts: Identify numbers + currency units (such as "yuan, ten thousand yuan, US dollars"), such as "5000 yuan" and "100,000 US dollars" are marked as "amounts"; For each entity object, record its starting and ending positions in the text (such as "Zhang" is located at the 5th - 6th characters in the 3rd paragraph), and associate the corresponding character coordinates.

[0047] Analyze the sentence components and grammatical relationships of the text content: Demarcate sentence boundaries: Use periods, question marks, or exclamation points to split sentences, such as "Party A is responsible for transportation. Party B is responsible for acceptance." into two sentences. Identify subject-verb-object structures: connect the subject and object through verbs, such as "Party A (subject) signed (verb) contract (object)", marking the subject-verb-object relationship of "Party A-signed-contract"; Identify modification relationships: For example, adjectives modify nouns ("red (attributive) file (central word)"), adverbs modify verbs ("immediately (adverbial) execute (predicate)"), and record the positional relationship between the modifier and the modified word; For complex sentences (such as those containing clauses), mark them in layers according to the “main clause-subordinate clause” structure, such as “Party A should pay the purchase price (main clause) after receiving the goods (time adverbial clause)”.

[0048] Mining the semantic logic between entities and sentences in text: Causal relationship: Identified by conjunctions (such as "because, so, cause"), such as "due to heavy rain (cause), delivery was delayed (result)"; Conditional relationships: Identified by words such as "if, as long as, unless", etc., such as "If Party B breaches the contract (condition), it must pay liquidated damages (result)"; Parallel relationship: Identified by words such as "and", "and", "at the same time", such as "Party A and Party B (parallel entities) bear joint responsibility"; Reference relationship: Identify the relationship between the pronoun and the antecedent, such as "Party A should pay on time, otherwise it (referring to Party A) will be held responsible"; For each semantic relationship, record the type, position and relationship strength of the associated entities (e.g., the strength of “causal relationship” is higher than that of “parallel relationship”).

[0049] Chapter level number extraction: Determine the chapter level based on the section markers and title font size differences in the preprocessed document: Identify title text: Filter text with a font size 1.5 times larger than the main text (e.g., main text 12pt, title ≥18pt) and left / center aligned, such as "Chapter 1 General Provisions of Contract"; Parsing level numbering: Determine the level depth according to the numbering rules such as "one, two, three...", "1.1, 1.2...", "(1), (2)...", etc. For example, "Chapter 1" is level 1, "1.1" is level 2, and "(1)" is level 3; Establish a hierarchical relationship: For example, if "1.1 Delivery Time" belongs to the sub-level of "Chapter 1 General Provisions of Contract", record the association of "Parent Number = Chapter 1, Child Number = 1.1".

[0050] Determine the attribution of paragraphs and chapters based on the paragraph position and content association: Sort the paragraphs by the vertical coordinate of the page and calculate the distance between the paragraph and the title (e.g. the vertical distance between the top edge of a paragraph and the bottom edge of "1.1 Delivery Time" = 30 pixels); Attribution rule: By default, a paragraph is attributed to the nearest upper title. If the distance is more than 200 pixels (about 2 lines of text height), it is attributed to the upper-level title. For example: The paragraph 30 pixels below "1.1 Delivery Time" belongs to "1.1", and the paragraph 250 pixels away from "1.1" belongs to "Chapter 1".

[0051] Identify the nesting relationship of list items based on the list symbol sequence and indentation position coordinates: Identify list symbols: such as “●, ○, ■” (bullet points), “1., 2., (a), (b)” (numbers), and use the type of symbol to mark each list item; Calculate the indent distance: Based on the left edge of the page, count the left indent pixels of the list item text (e.g., the first-level list is indented 30 pixels, and the second-level list is indented 60 pixels); Determine the nesting level: For every 30 pixels (about 1 character width) of indentation distance increase, the level increases by 1. For example, an indentation of 30 pixels is level 1, and 60 pixels is level 2. Record the nesting relationship of "list item A (level 1) contains list item a (level 2)".

[0052] Based on the table frame topology, analyze the logical association of cells: Identify header and content cells: Header cells are usually located in the first row or column of a table and are boldfaced. For example, "serial number, name, quantity" are marked as headers. Determine row and column associations: For example, if cell "1" (serial number) in row 2, column 1 corresponds to "Device A" (name) in row 2, column 2, record "Row Association: 1-Device A"; Identify merged cells: Identify by the interruption position of the table frame line. If there is no vertical dividing line in a certain area, it is a horizontal merged cell (such as A1-C1 merged). Record the merged range and the contents included. For nested tables, mark the ownership of the inner table cells and the outer table cells, such as "row 3, column 2 of the outer table contains the inner table."

[0053] Precise coordinate and spacing data provide the foundation for analyzing the document's physical structure, effectively distinguishing between text and non-text areas and identifying nested table relationships. This solves the problem of inaccurate element positioning caused by tilted scanning or complex layouts. The extraction of entity identifiers and semantic associations enables a deep understanding of document content, supporting key information extraction (such as contract amounts and dates) and semantic reasoning (such as causal relationship judgment), improving the accuracy of downstream tasks. The extraction of features such as chapter hierarchy and paragraph attribution clearly presents the document's organizational framework, visualizing the hierarchical relationships of complex structures such as nested lists and multi-level headings, facilitating structured data generation and intelligent review. The combination of spatial layout, semantics, and logical features overcomes the limitations of single-feature analysis and achieves a comprehensive analysis of "form, content, and structure."

[0054] In a preferred embodiment of the present invention, spatial layout features are used to construct a document spatial reference system based on text line tilt angle statistics and table frame corner coordinates. The text line start and end connection points, table physical boundary corners, and paragraph spacing center points are selected as spatial reference points. Spatial calibration parameters are generated based on the coordinate offsets between the reference points. The output of the calibrated spatial layout features may include: Calculate the text line tilt angle statistics based on the text line boundary coordinates and character spacing data in the spatial layout features; Based on the statistical values ​​of the text line tilt angle and the coordinates of the table frame corners, a Cartesian coordinate system with the upper left corner of the page as the origin is established; Generate a set of geometric calibration reference points based on the coordinates of the first and last characters of a text line, the coordinates of the outermost corner points of a table, and the coordinates of the center of the spacing between adjacent paragraphs in the spatial layout features; and generate rotation angle and translation distance parameters based on the coordinate offset of the geometric calibration reference point set in the Cartesian coordinate system. All coordinate data in the spatial layout feature are normalized by the rotation angle and translation distance parameters, and the calibrated spatial layout feature is output.

[0055] In an embodiment of the present invention, the statistical value of the text line tilt angle is calculated: Based on the text line boundary coordinates in the spatial layout features (the left, right, top, and bottom boundary pixel coordinates of each text line), traverse all text lines: For a single line of text, take the top point of the left boundary (left boundary x-coordinate, top boundary y-coordinate) and the bottom point of the right boundary (right boundary x-coordinate, bottom boundary y-coordinate). Calculate the tilt angle of the text line using the slope of the line connecting the two points (e.g., if the line slopes 3° from the top left to the bottom right, the angle is 3°). If the text line is horizontal (slope is 0), the tilt angle is 0°; if it tilts to the left (the line goes from the top right to the bottom left), the angle is negative (e.g., -2°). The tilt angles of all text lines are collected, and the average (e.g., if the sum of the angles of 100 lines is 300°, the average angle is 3°), median (the middle value after sorting the angles), and standard deviation (reflecting the dispersion of the angle distribution) are calculated. The average is used as the statistical value of the text line tilt angle.

[0056] Create a Cartesian coordinate system with the upper left corner of the page as the origin: A two-dimensional Cartesian coordinate system is constructed, with the top-left pixel of the page as the origin (0, 0), with the positive x-axis pointing horizontally to the right and the positive y-axis pointing vertically downward. The coordinate system orientation is calibrated using statistical values ​​for text row tilt angles and the coordinates of the table frame corners. If the statistical value for text row tilt angles is 3° (tilted to the right), the coordinate system is rotated 3° clockwise from the origin to align the x-axis with the text row baseline. If the table frame corner coordinates indicate a tilt (e.g., the top-left corner (100, 200) or the top-right corner (500, 210), with the connecting line not horizontal), the horizontal midline angle of the table frame is used as a secondary reference to ensure the coordinate system aligns with the overall orientation of the main elements (text, table) on the page. Once the coordinate system is established, the positions of all pixels are expressed as (x, y) coordinates. For example, the bottom-right pixel coordinates are (2480, 3508) (corresponding to A4 paper at 300 DPI).

[0057] Generate a set of geometric calibration reference points: Text line start and end connection points: For consecutive text lines (such as multiple lines of text in the same paragraph), take the lower right corner coordinates of the last character of the previous line of text and the upper left corner coordinates of the first character of the next line of text. These two points form the start and end connection points (such as "end of first line" (500, 220) and "beginning of second line" (100, 230)).

[0058] Table physical boundary corners: Select the four corners of the table's outermost border (upper left, upper right, lower left, and lower right). For example, the table's outer border corners are (200, 300), (600, 300), (200, 500), and (600, 500).

[0059] Paragraph spacing center point: Calculate the vertical spacing between two adjacent paragraphs (the difference between the y coordinate of the lower boundary of the previous paragraph and the y coordinate of the upper boundary of the next paragraph), and take the midpoint coordinate of the spacing as the center point (for example, if the spacing is 40 pixels, the midpoint y coordinate = (the y coordinate of the previous paragraph below + the y coordinate of the next paragraph above) / 2); finally, summarize the above points into a set of geometric calibration reference points, and mark each point with a type (such as "text connection point", "table corner point", "paragraph midpoint").

[0060] Generate rotation angle and translation distance parameters: Calculate the offset between the actual coordinates of the reference point set in the current coordinate system and the theoretical coordinates (the coordinates in an ideal state without tilt and offset): Rotation Angle: For the table corners, ideally, the line connecting the upper left and upper right corners should be parallel to the x-axis (angle 0°). If the actual angle of the line is 2°, the rotation angle parameter is -2° (correction by rotating 2° counterclockwise).

[0061] Translation distance: For the connection point between the first and last lines of text, ideally the two lines should be left-aligned (the x-coordinates of the first characters are the same). If the actual x-coordinates of the first characters differ by 5 pixels, the translation distance in the x direction is -5 pixels (5 pixels to the left). If the actual y-coordinate of the paragraph spacing center point is 10 pixels higher than the theoretical value, the translation distance in the y direction is -10 pixels (10 pixels upward).

[0062] The offsets of all reference points are combined and the average value is taken as the final rotation angle (such as -1.5°) and translation distance (such as -3 pixels in the x direction and -8 pixels in the y direction).

[0063] Normalize the spatial layout features: Based on the generated rotation angle and translation distance parameters, all coordinate data (text line boundaries, character spacing, table corners, non-text area positions, etc.) of the spatial layout feature are adjusted: Rotation processing: Rotate each coordinate point according to the rotation angle (such as -1.5°) to make the text lines and table borders parallel to the x-axis and eliminate the overall tilt.

[0064] Translation processing: Move the rotated coordinate point by the translation distance (-3 pixels in the x direction, -8 pixels in the y direction) to align the reference point with the theoretical position (for example, the upper left corner of the table is adjusted from (200, 300) to (197, 292)).

[0065] Normalization check: After adjustment, recalculate the offset of the reference point to ensure that the rotation deviation of all points is ≤0.5° and the translation deviation is ≤2 pixels. Otherwise, reoptimize the parameters and repeat the process.

[0066] Output calibrated spatial layout features, including corrected text line boundary coordinates, character spacing data, table frame corner coordinates and non-text area positions. All coordinates are based on a unified Cartesian coordinate system to ensure consistency of spatial position.

[0067] By counting text line tilt angles and calibrating table corners, coordinate deviations caused by document scanning tilt and camera offset are eliminated, reducing the spatial position error of elements such as text and tables to ≤2 pixels. A Cartesian coordinate system and a set of geometric reference points, with the upper left corner of the page as the origin, standardize the spatial positions of different types of elements (text lines, tables, and paragraphs), resolving the issue of inconsistent coordinates across regions. After normalization, all coordinate data in the spatial layout features is based on the same scale and orientation, ensuring accurate spatial matching between semantic features and logical structure features. For complex elements such as nested tables and multi-level lists, the calibrated spatial coordinates clearly reflect their physical boundaries and hierarchical relationships, providing a precise spatial basis for identifying logical inclusion and juxtaposition between elements.

[0068] In a preferred embodiment of the present invention, semantic features, logical structure features, and calibrated spatial layout features are collaboratively analyzed to generate multimodal association features through mapping verification between semantic content and spatial location, and matching association between logical structure and spatial distribution. The multimodal association features may include: Based on the text line boundary coordinates and non-text area positions in the calibrated spatial layout features, the spatial consistency between the position coordinates of the entity objects in the semantic features and the actual coverage area of ​​the text lines is verified, and a mapping verification relationship between semantic content and spatial position is generated. Based on the table frame corner coordinates and paragraph distribution positions in the calibrated spatial layout features, the spatial distribution range of chapter level numbers and title font size differences in the logical structure features are associated with the spatial topological relationship of the list nesting rules to generate a matching association relationship between the logical structure and the spatial distribution. The mapping verification relationship between semantic content and spatial location and the matching association relationship between logical structure and spatial distribution are integrated to generate multimodal association features including semantic location verification identifiers and logical space topology identifiers.

[0069] In an embodiment of the present invention, the text positions (paragraphs, character indexes) of all entity objects (such as "Party A", "October 1, 2023", and "5,000 yuan") are obtained from semantic features, and their character coordinates recorded in the preprocessed document are associated (such as the character coordinates corresponding to "Party A" are (150, 300, 180, 330)).

[0070] Text line coverage matching: Based on the calibrated spatial layout features, extract the boundary coordinates of the text line where the entity object is located (e.g., left = 100, right = 500, top = 290, bottom = 340), and determine whether the coordinates of the entity object are completely contained within the text line boundaries: If the upper left x of the entity object ≥ the left boundary of the text line, the upper right x ≤ the right boundary of the text line, and the upper and lower y coordinates are within the upper and lower boundaries of the text line, it is determined as "spatially consistent"; If part of the entity object exceeds the text line boundary (e.g., x1 = 95 of the character "A" < the left boundary 100 of the text line), calculate the exceeding ratio (number of exceeding pixels / total number of entity pixels). If the ratio < 5%, it is still determined as "basically consistent", otherwise it is marked as "spatially inconsistent".

[0071] Non-text area conflict detection: Check whether the coordinates of the entity object overlap with non-text areas (such as pictures, blank areas): If the overlapping area between the entity coordinates and the image area (such as (300, 200, 500, 400)) > 10%, it is marked as "conflicting with non-text area"; If there is no overlap or the overlapping area < 10%, it is determined as "no conflict".

[0072] Generate mapping verification relationships: Add semantic position verification identifiers for each entity object, including "spatial consistency" (consistent / basically consistent / inconsistent) and "non-text area conflict" (yes / no), such as "Party A: spatially consistent, no conflict", "5000 yuan: basically consistent, no conflict".

[0073] According to the calibrated paragraph distribution positions, determine the spatial range corresponding to each chapter title: The spatial range of the chapter title is its text line boundary coordinates (e.g., "1.1 Delivery Terms" corresponds to (100, 500, 200, 530)); The spatial range of the chapter content is the area from the lower boundary of the title to the upper boundary of the next chapter title (e.g., from y = 530 to y = 1000).

[0074] Association between title font and spatial distribution: Compare the correspondence between the differences in title font sizes and spatial positions in the logical structure features: The first-level title (20pt) should be located in the upper part of the page and have a small left indentation (e.g., x = 50), and the second-level title (16pt) has a larger left indentation (e.g., x = 80) and is located below the first-level title. If this rule is met, it is marked as "font - space matching"; If the second-level title is located above the first-level title or the left indentation is less than that of the first-level title, it is marked as "font - space mismatch".

[0075] Verification of table and chapter affiliation: Based on the coordinates of the table frame corners (e.g., (200, 600, 600, 800)), determine whether the table is within the spatial range of a certain chapter content (e.g., y=530 to y=1000): If the upper boundary of the table ≥ the upper boundary of the chapter content and the lower boundary ≤ the lower boundary of the chapter content, then it is determined that the "table affiliation is correct" (e.g., it belongs to "1.1 Delivery Terms"); if the table spans chapter ranges (e.g., lower boundary y=1050>chapter content lower boundary y=1000), it is marked as a "cross-chapter table" and needs to be further associated with the adjacent chapter logic.

[0076] Based on the list nesting level (first-level lists are indented 30 pixels, second-level lists are indented 60 pixels), check whether the spatial position of the list items complies with the topological rules: The left boundary x of the second-level list item should be greater than the left boundary x of the first-level list item (for example, first-level x=100, second-level x=130), and it should be located below the first-level list item (with a larger y coordinate). If this is the case, it is marked as "nesting topology correct." If the left boundary x=90 of the second-level list item is less than x=100 of the first-level list item, it is marked as "nesting topology incorrect."

[0077] Generate matching associations: Summarize the above verification results and add a logical space topology identifier to each logical structure element, such as "Section 1.1: Font-Space Matching", "Table 1: Correct Attribution", and "List Item 2.1: Correct Nesting Topology".

[0078] Fusion generates multimodal correlation features: Set the fusion priority: "Spatial inconsistency" and "Non-text area conflict" in the semantic position verification mark are high priority, and "Attribution error" and "Topology error" in the logical space topology mark are the second highest priority.

[0079] Cross-feature association mapping: Associate the entity object in the semantic feature (such as "Delivery Date: October 1, 2023") with the chapter in the logical structure feature (such as "1.1 Delivery Terms") through spatial coordinates: If the entity coordinates are within the chapter content range and both are marked "spatial consistency" and "correct attribution", an "entity-chapter" association is established; associate the entity in the table (such as "unit price: 100 yuan") with the logical attribution of the table (such as "2.2 Cost Details"). If the table attribution is correct and the entity space is consistent, a "table entity-table chapter" association is established.

[0080] Multimodal feature integration: The multimodal association features finally generated include: the semantic content, spatial coordinates, semantic position verification identifier and logical structure (chapter / list / table) of each entity object; the hierarchical number, spatial range, logical space topology identifier and list of entity objects included in each logical structure element; cross-element association relationships (such as "entity A belongs to chapter B" and "table C contains entity D").

[0081] Through spatial location verification and logical topology matching, the disconnect between semantic entities and physical locations, and the inconsistency between logical structure and spatial distribution, are resolved. For complex structures like nested tables and multi-level lists, multimodal association features clearly present the triple association of "semantic content, spatial location, and logical hierarchy," improving the accuracy of hierarchical recognition. The abstract meaning of semantic features, the physical location of spatial features, and the structural relationships of logical features are integrated into a complete knowledge system, avoiding parsing errors caused by incomplete information from a single modality.

[0082] In a preferred embodiment of the present invention, multimodal association features are input into a pre-trained visual-linguistic multimodal large model, and through combined semantic understanding and structural analysis, parent-child hierarchical relationships and semantic type identifications of document elements are output, which may include: The multimodal correlation features are input into the pre-trained visual-linguistic multimodal model, where the following operations are performed: Based on the entity object identification and syntactic structure relationship in the semantic features, combined with the title font size difference and chapter level number in the logical structure features, the semantic type identification and spatial distribution range of the chapter title are determined; Based on the text line boundary coordinates and paragraph distribution positions in the calibrated spatial layout features, the spatial inclusion relationship between the spatial distribution range of the chapter title and the actual coverage area of ​​the paragraph is verified, and the parent-child inclusion relationship at the chapter level is established; Based on the semantic association relationship in the semantic features and the table frame topology relationship in the logical structure features, the semantic association and physical boundary nesting relationship between cells are identified, and the hierarchical structure of the nested table is established; Based on the list symbol sequence rules and indentation position coordinates in the logical structure features, the syntactic structure relationship of the list items in the semantic features is matched to establish the nested hierarchical relationship of the list items; Finally, the semantic type identification and parent-child hierarchical relationship of each document element are output; the semantic type identification includes five types of elements: chapter title, paragraph body, table, list item, and picture, and the parent-child hierarchical relationship includes three groups of attributes: parent element identification, child element identification, and hierarchical depth.

[0083] In this embodiment of the present invention, the visual-language multimodal large model adopts a three-level structure of "visual encoding part-language encoding part-decoding part" to achieve cross-modal feature extraction, fusion and reasoning output: The visual encoding part (based on a modified ResNet-50 network) is specifically designed to extract spatial layout features from document images. It takes a fixed-resolution (224×224 pixel) document image as input and outputs a 512-dimensional spatial feature vector. The specific design is as follows: The algorithm consists of five groups of convolutional structures connected in series, each containing three convolutional layers (with kernel sizes of 7×7, 3×3, and 3×3, respectively), batch normalization (used to stabilize data distribution during training), and the ReLU activation function (f(u)=max(0,u), where f(u) is the output value after ReLU activation; u represents the input feature value (i.e., the numerical result output by the convolutional layer or the previous layer, which may be positive, negative, or zero). This nonlinear transformation enhances the nonlinear expressiveness of features. Groups 1-3 of the convolutional structures focus on extracting local detail features: using convolution operations to capture pixel gradient changes at text line boundaries (for example, the 0-255 pixel value jumps caused by light and dark contrast at character edges), rectangular features of table corners (such as the horizontal and vertical pixel value mutation pattern at the upper left corner (x1, y1)), and contour edge features of image regions. The fourth and fifth convolutional structures are responsible for integrating global structural features. They compress local features through a maximum pooling operation (pooling kernel size 2×2) to generate a 2048-dimensional feature vector containing global information such as text line orientation (horizontal or tilted), table outlines (rectangular boundary integrity), and paragraph density. Finally, a fully connected layer compresses this 2048-dimensional vector to 512 dimensions, which serves as the standardized output of the image spatial features to ensure compatibility with the dimensionality of subsequent language features.

[0084] Language encoding (customized based on the BERT model): This part processes the semantic and logical features in document text. It takes a structured text sequence as input and outputs a 768-dimensional semantic feature vector. The specific processing flow is as follows: The input is a text sequence containing semantic features and logical structure features (such as "Entity: 1.1 Payment Terms; Font: 16pt; Numbering Rule: Chapter > Section"). First, it is split by the WordPiece tokenization method (for example, "Payment Terms" is split into subwords such as "Pay", "ment", "ter", "ms", etc.) and mapped to a vocabulary index of 30,522 dimensions. The tokenized results are embedded: the vocabulary index is converted into a 768-dimensional word embedding vector, and at the same time, position encoding (marking the position of each subword in the sequence, such as the position encoding of the first word is [1, 0, 0,...], the second is [0, 1, 0,...]) and type encoding (distinguishing semantic feature and logical feature fields) are added. Deep semantic modeling is performed through a 12-layer Transformer encoding structure: each layer contains 8-head self-attention mechanism (calculating the association weights between each subword and other subwords in the sequence, for example, the attention weight between "1.1" and "Payment Terms" is 0.8, and the weight with "Font" is 0.2), and a feed-forward neural network (enhancing the non-linear transformation ability of features); finally, a 768-dimensional text semantic feature vector is output, containing information such as the type identification of entity objects (such as "Chapter Title"), syntactic structure relationships (such as the "Title - Content" pairing), and hierarchical numbering rules (such as "1.1 belongs to the first chapter").

[0085] The decoding part (designed based on the Transformer structure): responsible for fusing visual and language features and outputting the inference results, consisting of 6 decoding structures, each layer containing cross-attention processing, self-attention processing, and a feed-forward neural network: Cross-attention processing: The 512-dimensional features output by the visual encoding component serve as keys and values, while the 768-dimensional features output by the language encoding component serve as queries. These three sets of features are first dimensionally adapted, using a linear transformation to unify the 512-dimensional visual features and the 768-dimensional language features to the same dimension (e.g., 64 dimensions per head, 8 heads total, 512 dimensions). Next, the similarity between the query and the key is calculated: a dot product is performed on the query vector (64 dimensions) and the key vector (64 dimensions) for each head to obtain a raw relevance score (e.g., the dot product of the query vector corresponding to the word "table" with the key vector corresponding to the table corner coordinates is 8.6). This score is then scaled by dividing it by the square root of the feature dimension to obtain a scaled relevance score. All scaled relevance scores are normalized, using numerical transformations to ensure that the sum of the relevance probabilities for all keys corresponding to the same query is 1 (e.g., the relevance probabilities for the word "table" with table corners, text line boundaries, and image outlines are 0.82, 0.15, and 0.03, respectively, after transformation). Finally, the normalized probability is weightedly summed with the corresponding value vector. That is, each value vector is assigned a weight according to its associated probability and accumulated to obtain a feature vector that integrates the "text word-spatial position" association information (for example, the feature vector of the word "table" assigns 82% of the weight to the spatial features of the table corners and 15% to the text line boundary features), thereby achieving deep fusion of cross-modal features.

[0086] Self-attention processing: For the 1024-dimensional features obtained after fusion, the correlation strength between each element within the feature is quantified to strengthen the logical correlation between features. The specific process is as follows: First, the 1024-dimensional features are split into three vectors of equal length, serving as the query, key, and value. For example, subvectors related to "chapter title" are extracted from the fused features as the query, and subvectors related to other elements such as "paragraph" and "table" are extracted as the key and value. The matching degree between the query and key is calculated by performing a dot product between each query vector and all key vectors to obtain a raw association score (e.g., the dot product of the query vector corresponding to "chapter title" with the key vector corresponding to a paragraph is 7.2, and the dot product of the query vector corresponding to the key vector corresponding to a table is 3.5). This score reflects the closeness of the association between different elements. These raw association scores are scaled (divided by the square root of the feature dimension) to avoid excessively large values ​​that reduce the discriminability of the association degree. Numerical conversion is then performed so that the sum of the association probabilities for all keys corresponding to the same query is 1 (e.g., the association probability of "chapter title" with a paragraph is 0.7, the association probability with a table is 0.2, and the association probability with all other elements is 0.1). These normalized association probabilities are weighted and summed with the corresponding value vectors. That is, value vectors like "paragraph" and "table" are weighted according to their association probabilities with "section title," and the sum is accumulated to form a new feature vector. This process quantifies the hierarchical dependencies between elements like "section title" and "paragraph" and encodes them into the features, enhancing the logical connections between features (such as inclusion and subordination). Finally, a Softmax layer outputs the probability distribution of semantic type identification (e.g., the probability of "section title" is 0.95) and the predicted parent-child hierarchical relationships.

[0087] To ensure the generalization ability and inference accuracy of the model, the training data collection and preprocessing process is as follows: We collected 100,000 documents in various formats (PDF, Word, scans, and image-based documents) and fields (contracts, reports, papers, and forms). 30% of these documents were scanned (including samples with varying quality, such as skew, blur, and shadows), to cover the diverse document types that might be encountered in real-world applications. We used automated annotation tools to annotate the dataset with features, covering three key areas: Semantic type: Mark the type and attributes of each document element, such as "Chapter title (level 1, font 20pt, number 'Chapter 1')", "Table (outer layer, header 'Expense Details')", "List item (level 2, symbol '○')".

[0088] Spatial coordinates: Mark text line boundaries with pixel accuracy (e.g., "Left 100" means the left edge of the text line is 100 pixels away from the starting point of the upper left corner of the document image in the horizontal direction, "Right 500" means the right edge of the text line is 500 pixels away from the starting point in the horizontal direction, "Top 200" means the top edge of the text line is 200 pixels away from the starting point in the vertical direction, "Bottom 220" means the bottom edge of the text line is 220 pixels away from the starting point in the vertical direction), table corners (e.g., (100, 200) is the upper left corner, (600, 800) is the lower right corner), and paragraph spacing centers (e.g., (300, 250) is the midpoint between two adjacent paragraph spacings).

[0089] Hierarchical relationship: marked with the triple of "parent element ID-child element ID-depth". For example, "CH-001 (Chapter 1)-CH-002 (1.1)-2" means that "1.1 Payment Terms" is a sub-chapter of "Chapter 1 Contract Subject Matter" and the hierarchical depth is 2.

[0090] Data preprocessing: Image data: The image is uniformly scaled to 224×224 pixels, and the scans are tilted (rotated to a horizontal state) and denoised (speckle noise removed) to ensure consistency of image features.

[0091] Text data: Convert to lowercase, remove unnecessary spaces, special characters, and other interfering information, and use the [SEP] symbol to separate different features (for example, "1.1 Payment Terms [SEP] 16pt [SEP] Left 100"), which means "The text '1.1 Payment Terms' uses a 16-point font size and has a left indent of 100 units." The purpose of [SEP] is to clearly separate the three different types of information: "text content," "font size," and "left indent."

[0092] Dataset division: The dataset is divided into a training set (80,000 samples), a validation set (10,000 samples), and a test set (10,000 samples) in a ratio of 8:1:1. Data augmentation is performed on the training set, including randomly cropping 10% of the image area and adding a ±5° rotation change to improve the model's anti-interference ability.

[0093] The training process of the large visual-language multimodal model involves multiple rounds of iterative parameter optimization to ensure that it can accurately predict the semantic types and hierarchical relationships of document elements. The visual encoding part loads the weights of ResNet-50 pre-trained on the ImageNet dataset, leveraging its learned image feature extraction capabilities. The language encoding part loads the weights of pre-trained BERT-base, reusing its text semantic understanding capabilities. The parameters of the decoding part use the Xavier initialization method to ensure that the variance of the input and output is consistent, avoiding gradient explosion or vanishing in the early stages of training.

[0094] Training details: Each batch of 32 documents is fed with image and text features, along with the annotated semantic types and hierarchical relationships as supervisory signals. A joint loss function is used: semantic type prediction uses cross-entropy loss (calculating the probability difference between the model's predicted type and the annotated type), and hierarchical relationship prediction uses triplet loss (optimizing the distance between correct "parent-child" element pairs to be smaller than incorrect pairs, enhancing the discriminability of hierarchical relationships). The AdamW optimizer is used, with an initial learning rate of 2e-5 and a weight decay coefficient of 0.01 (to prevent overfitting by suppressing excessive weights). Each training round consists of 2500 steps (80,000 training data points divided by 32 documents / batch = 2500 steps), for a total of 100 rounds. After each round, accuracy is calculated on the validation set (semantic type prediction accuracy = number of correct predictions divided by total number of predictions; similarly for hierarchical relationship accuracy), to dynamically monitor model performance. When the validation set accuracy remains above 90% for five consecutive rounds (for example, the accuracy rates for rounds 60, 61, 62, 63, 64, 65, 66, 67, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 77, 78, 79 ...9, 81, 82, 83, 84, 85, 86, 77, 79, 81, 82, 83,

[0095] Convert the multimodal correlation features into a format that can be directly processed by the large visual-linguistic multimodal model as follows: Visual input: A calibrated spatial layout feature map is used, in which key locations such as text line boundaries and table corners are marked with pixels (for example, a 255-pixel value is used to mark the left boundary of a text line, and a 128-pixel value is used to mark the upper left corner of a table). The image resolution is uniformly set to 224×224 pixels to ensure that it matches the input requirements of the visual encoding part.

[0096] Language input: Semantic features and logical structure features are organized into a text sequence (for example, "Entity: 1.1 Payment Terms; Syntax: Title + Content; Font: 16pt; Number: 1.1"). This is then converted into a 512-token sequence through word segmentation and padding (with [PAD] used to fill in any missing tokens), meeting the input format requirements of the language encoding part.

[0097] Input synchronization: Visual input and language input are aligned through timestamps and synchronously fed into the visual-language multimodal large model, triggering the model's joint semantic understanding and structural analysis process.

[0098] The model extracts the entity object identifier "1.1 Payment Terms" from the language input. The BERT encoder analyzes its syntactic structure ("number + keyword" conforms to the title sentence structure) and extracts the logical features of font size (16pt) and number (1.1). The visual encoder locates the boundary coordinates of the text line containing the entity (100° left, 300° right, 200° top, 220° bottom). The cross-attention layer calculates the probability of association between "1.1" and the "section title" label (0.96). This probability exceeds the threshold of 0.8, and the entity is labeled "Level 2 Section Title." Based on the text line boundary coordinates, the spatial extent is determined to be a rectangular region (100° left, 300° right, 200° top, 220° bottom). The center coordinates are calculated as ((100 + 300) / 2, (200 + 220) / 2) = (200, 210).

[0099] Paragraph spatial range extraction: The visual encoder identifies three text lines in a paragraph based on spatial layout features. The boundaries are (left 100, right 500, top 230, bottom 250), (left 100, right 500, top 255, bottom 275), and (left 100, right 500, top 280, bottom 300). The minimum rectangle containing these three lines is calculated as (left 100, right 500, top 230, bottom 300) and used as the paragraph spatial range.

[0100] Spatial inclusion verification: The model calculates the positional relationship between the chapter title ("Left 100" refers to the horizontal pixel position of the left edge of the chapter title, "Right 300" refers to the horizontal pixel position of the right edge of the chapter title, "Top 200" refers to the vertical pixel position of the top edge of the chapter title, and "Bottom 220" refers to the vertical pixel position of the bottom edge of the chapter title) and the paragraph: If the paragraph's upper margin (230) is ≥ the chapter's lower margin (220), and the paragraph's left margin (100) and right margin (500) are aligned with the chapter's left margin, while the right margin (500) exceeds the chapter's right margin (300), but with a margin error of 200 pixels (exceeding ±10 pixels), the paragraph's content is further examined for semantic association with the chapter title ("Payment Terms" is related to "Payment Method" in the paragraph), ultimately determining that the paragraph belongs to that chapter. For the multi-level sections "1.1 Payment Terms" and "Chapter 1 Contract Subject Matter," the lower margin (500) of "1.1" is verified to be ≤ the lower margin (1000) of "Chapter 1," and the left indent (120 pixels) of "1.1" is greater than the left indent (100 pixels) of "Chapter 1," establishing a parent-child containment relationship.

[0101] Hierarchical depth calculation: "Chapter 1" is a level 1 chapter (depth 1), "1.1" is its sub-chapter, depth = 1 + 1 = 2; the depth of the paragraph to which it belongs is the same as "1.1", that is, 2.

[0102] Feature extraction and nested identification: The model extracts the boundary coordinates of Table A (left 100, right 600, top 400, bottom 800) and Table B (left 200, right 500, top 450, bottom 750) from logical features. Visual features indicate that Table B is completely contained within Table A. Semantic features extract the association between Table A's header "Expense Details" and Table B's content "Equipment Unit Price: 100 yuan" ("Details" and "Unit Price" have an inclusion relationship), determining that Table B is a child table of Table A. The model identifies the small table (left 210, right 290, top 510, bottom 590) within the cell (left 200, right 300, top 500, bottom 600) as a "nested table within a cell," since the small table's boundaries are completely within the cell. The parent element is Table A.

[0103] Hierarchical records: The depth of Table A is 1, the depth of Table B is 2, and the depth of the small table is 3. The record association relationship is "Table A contains Table B (general table-detailed table), and Table B contains the small table (detailed table-item table)."

[0104] Feature matching and nesting verification: The model extracts list item A (symbol "●", 30 pixels indented to the left, content "Equipment List") and list item B (symbol "○", 60 pixels indented to the left, content "Server: 1 unit").

[0105] Calculate the indentation difference: 60 - 30 = 30 pixels. The symbol types are different, and the content of B is a detailed description of A. Visual features verify that B's upper boundary (350) ≥ A's lower boundary (340), and its left boundary (130) > A's left boundary (100), determining that B is a sublist item of A.

[0106] Hierarchical record: List item A has a depth of 1, list item B has a depth of 2, and the association relationship is "A contains B".

[0107] Type identification assignment: Traverse all elements, mark "Chapter 1" as the "Level 1 Chapter Title", "1.1 Payment Terms" as the "Level 2 Chapter Title", the paragraph to which it belongs as the "Paragraph Body", Table A as the "Outer Table", Table B as the "Inner Table", List Item A as the "Level 1 List Item", List Item B as the "Level 2 List Item", and the "Equipment Diagram" in the document as the "Picture" (associated with the paragraph "Equipment Specifications").

[0108] Add attributes to each element: Parent element identification: the parent element of the paragraph is "1.1 Payment Terms", and the parent element of Table B is "Table A"; Sub-element identification: The sub-elements of "Chapter 1" are "1.1 Payment Terms" and "1.2 Delivery Terms"; Level depth: "1.1 Payment Terms" depth 2, Table B depth 2, List Item B depth 2.

[0109] Final output: Structured data containing the type, coordinates, parent / child identifiers, and hierarchical chains of all elements, such as "Section 1 (depth 1) → Section 1.1 (depth 2) → Paragraph 1 (depth 2) → Table A (depth 2) → Table B (depth 3)".

[0110] Through joint analysis of multimodal features, the problem of type confusion for complex elements such as nested tables and multi-level lists is resolved (e.g., the probability of misidentifying sublist items as paragraphs is reduced), and the accuracy of hierarchical relationship recognition is improved. By integrating semantic, spatial, and logical features, the parent-child relationship of document elements is based not only on position but also on semantic rationality (e.g., the "Payment Amount" paragraph is accurately associated with the "Payment Terms" section), avoiding the erroneous associations caused by purely spatial analysis. The output type identification and hierarchical relationships provide a direct basis for the subsequent construction of a tree-like semantic structure, ensuring that downstream tasks (such as intelligent review and data warehousing) can directly access structured information, reducing secondary processing costs. Joint analysis based on pre-trained large models can adapt to the structural differences of documents in different formats (such as contracts, reports, and forms), and improve the robustness of parsing non-standard layouts (such as irregular indentations and tables that span multiple pages).

[0111] In a preferred embodiment of the present invention, based on the parent-child hierarchical relationship, a tree semantic structure is constructed with chapters as parent units, paragraphs as child units, and nested tables and list items as terminal units, and the tree semantic structure is converted into a structured digital output, which may include: The semantic type identifier and parent-child hierarchical relationship are used as input sources; when the semantic type identifier is a chapter title, a chapter root node is created according to the parent element identifier in the parent-child hierarchical relationship; when the semantic type identifier is a paragraph body, it is mounted under the corresponding chapter node according to the parent element identifier in the parent-child hierarchical relationship; when the semantic type identifier is a table, cell child nodes are recursively constructed according to the hierarchical depth value in the parent-child hierarchical relationship, and linked to the corresponding chapter or paragraph node; when the semantic type identifier is a list item, based on the hierarchical depth value and parent element identifier in the parent-child hierarchical relationship, a list item node chain is established and mounted to the corresponding paragraph node; when the semantic type identifier is a picture, it is linked to the corresponding paragraph or chapter node according to the parent element identifier in the parent-child hierarchical relationship; finally, a tree-like semantic structure including node type, semantic content, parent element identifier, child element identifier and hierarchical depth is generated, and the tree-like semantic structure is converted into a standard structured data format for output.

[0112] In this embodiment of the present invention, the system uses semantic type identifiers (such as "section title," "paragraph body," and "table") and parent-child hierarchical relationships (including each element's parent element identifier, child element identifier, and hierarchical depth) as core input to build a complete element information index table. In this index table, each document element is associated with a unique identifier ID, along with key information such as its semantic content (such as the text of a section title or the specific content of a paragraph), its spatial location coordinates within the document, and its hierarchical depth.

[0113] Construct the chapter root node: When the semantic type is identified as "Chapter Title", the system will first filter out the chapter titles with a hierarchical depth of 1 (such as "Chapter 1 Basic Terms of the Contract") and use it as the root node of the entire tree-like semantic structure.

[0114] Each root node is assigned a unique ID (for example, "Chapter-001"), and the node type is clearly recorded as "Chapter Title," with the semantic content being the specific text of that chapter (for example, "Basic Terms of the Contract"). Since the root node is the top level of the tree structure, its "Parent Element Identifier" is marked as "None," while its "Child Element Identifier" is temporarily empty, and the hierarchical depth is fixed at 1. If a document contains multiple top-level chapters (for example, both "Preface" and "Chapter 1"), the order of the root nodes is determined by the order in which these chapters appear in the document (arranged from top to bottom based on their vertical coordinate positions on the page).

[0115] Mount the paragraph body node: For elements with a semantic type identifier of "paragraph body", the system will mount them under the corresponding chapter node according to the "parent element identifier" in the parent-child hierarchical relationship. For example, if the parent element identifier of a paragraph is "Chapter-001", the system will find the chapter node corresponding to this identifier in the index table (i.e., "Chapter 1 Basic Terms of the Contract"). A unique ID is assigned to this paragraph (such as "Para-001"), and the node type is marked as "paragraph body". The semantic content is the entire text of the paragraph (such as "Party A shall pay the advance payment within 30 days after the signing of the contract"). At the same time, the "parent element identifier" of the paragraph is recorded as "Chapter-001", and the "child element identifier" is empty (the paragraph does not contain a lower-level node). The hierarchical depth is consistent with the chapter to which it belongs (such as level 1). Finally, the paragraph node is added to the "child element identifier" list of the corresponding chapter node, forming a direct subordinate relationship of "chapter → paragraph".

[0116] Recursively construct table and cell nodes: When the semantic type is identified as "table", the system will recursively construct the table node and its subordinate cell nodes based on the "hierarchical depth" and "parent element identifier" in the parent-child hierarchical relationship, and link them to the corresponding chapters or paragraphs.

[0117] First, create a main node for the table and assign it an ID (e.g., "Table-001"). The tag type is "Table," and the semantic content is the table title (e.g., "Payment Schedule"). The "Parent Element Identifier" determines the chapter or paragraph to which it belongs (e.g., "Para-003" indicates that the table belongs to paragraph 3). The hierarchical depth of the table main node is the parent element depth plus 1 (e.g., if the parent element depth is 2, the table depth is 3). Next, parse the table's row and column structure (e.g., 5 rows and 3 columns), creating a child node for each cell. This node is assigned an ID (e.g., "Table-001-Cell-02-03" for row 2, column 3), a type of "Table Cell," and the semantic content is the text within the cell (e.g., "5000 Yuan"). The cell's "Parent Element Identifier" is the main node ID of the table to which it belongs ("Table-001"), and the hierarchical depth is the main node depth plus 1 (e.g., 4). If a cell contains a nested sub-table, repeat the above steps: create a main node for the sub-table (such as "Table-002"), set the parent element identifier to the ID of the cell to which it belongs ("Table-001-Cell-02-03"), set the depth to 5, and continue to create nodes for the cells of the sub-table. Finally, mount the table main node to the "Sub-element Identifier" list of the chapter or paragraph to which it belongs, forming a nested structure of "chapter / paragraph → table → cell → sub-table".

[0118] Create a list item node chain: For elements with a semantic type identified as "list item", the system constructs a list item node chain based on the "hierarchical depth" and "parent element identifier" in the parent-child hierarchical relationship and mounts it under the corresponding paragraph node.

[0119] First, process the first-level list item: assign it an ID (such as "List-001"), a type of "list item," semantic content of the list item text (such as "Equipment Transportation Responsibility"), a parent element identifier of the paragraph to which it belongs (such as "Para-005"), and a hierarchy depth of the parent paragraph depth plus 1 (such as 2). If a second-level list item exists (that is, the list item is a child of the first-level list item), assign it an ID (such as "List-001-01"), semantic content of "Transportation Mode: Road Transport," a parent element identifier of the first-level list item ("List-001"), a hierarchy depth of 3, and attach it to the "Child Element Identifier" list of the first-level list item. Similarly, third-level list items are attached to second-level list items, forming a chain structure of "paragraph → first-level list item → second-level list item → third-level list item," ensuring that each child list item is one level deeper than the parent list item.

[0120] Associated image node: When the semantic type is identified as "Picture," the system links the image node to its corresponding paragraph or chapter node based on the "Parent Element Identifier" in the parent-child hierarchical relationship. The image node is assigned an ID (such as "Img-001"), the type is "Picture," the semantic content is the image's description (such as "Device Appearance Diagram"), and the parent element is the ID of the corresponding paragraph or chapter (such as "Para-004"). The hierarchical depth remains the same as the parent element (such as 2). The image node is added to the parent element's "Child Element Identifier" list, coexisting with nodes such as paragraphs and tables (for example, a chapter's child elements may contain paragraphs, tables, and images).

[0121] Integrate the tree structure and output structured data: The system aggregates all nodes to form a complete tree-like semantic structure: with the top-level chapter as the root node, the lower-level nodes are nested in sequence according to the hierarchy of "chapter → paragraph → table / list item / picture → cell / sub-list item", and each node contains complete attributes (node ​​type, semantic content, parent element identifier, child element identifier, hierarchical depth). The tree-like semantic structure is converted into standard structured data formats such as JSON and XML. For example, in the JSON format, each node is presented as a key-value pair, and the IDs of its subordinate nodes are recorded through the "children" field, clearly reflecting the nested relationship (such as the "children" of the chapter node contains the IDs of the paragraphs and tables to which it belongs, and the "children" of the table node contains the IDs of its cells). The final output structured data can be directly connected to downstream systems (such as databases and knowledge graphs).

[0122] Through the tree-like semantic structure, scattered chapters, paragraphs, tables and other elements are logically linked to clearly present the hierarchical framework of the document, solving the problem of unstructured documents being difficult to reuse directly. For complex structures such as nested tables and multi-level lists, the internal relationships are accurately reflected by recursively building node chains, so that the detailed information of the document (such as the relationship between table cells and sub-tables) can be fully preserved. The converted standard structured data can be directly called by intelligent review, data storage and other systems, avoiding the process of repeated document parsing. Each node records the parent / child element identifier and hierarchical depth, which can quickly locate the position of the element in the document and the associated content (such as the chapter to which a table belongs, the paragraph corresponding to a list item), facilitating information traceability and verification.

[0123] An embodiment of the present invention further provides a computing device comprising: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, executes the system described above. All implementations in the above system embodiments are applicable to this embodiment and can achieve the same technical effects.

[0124] The embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, causes the computer to execute the system described above. All implementations in the above system embodiments are applicable to this embodiment and can achieve the same technical effects.

[0125] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A multimodal document content cross-platform parsing system, characterized by: include: The document preprocessing module is used to receive multi-source heterogeneous document input data and generate preprocessed documents through format conversion, page segmentation, noise reduction and optical character recognition; Feature extraction module, which is used to extract three types of features based on preprocessed documents, including spatial layout features, semantic features, and logical structure features; The feature calibration module is used to use spatial layout features to construct a document spatial reference system based on the statistical values ​​of text line tilt angles and the coordinates of the table frame corners. The module selects the connection points of text lines, the physical boundary corners of tables, and the center points of paragraph spacing as spatial reference points, and generates spatial calibration parameters based on the coordinate offsets between the reference points. The module then outputs the calibrated spatial layout features. The feature association module is used to collaboratively analyze semantic features, logical structure features, and calibrated spatial layout features. It generates multimodal association features by mapping and verifying semantic content and spatial location, and matching and associating logical structure with spatial distribution. The type recognition module is used to input multimodal correlation features into a pre-trained visual-linguistic multimodal large model, and output the parent-child hierarchical relationship and semantic type identification of document elements through combined semantic understanding and structural analysis; The data generation module is used to construct a tree-like semantic structure based on the parent-child hierarchical relationship and semantic type identification, with chapters as parent units, paragraphs as child units, and nested tables and list items as terminal units, and convert the tree-like semantic structure into structured data output.

2. The multimodal document content cross-platform parsing system according to claim 1, characterized in that: Receive multi-source heterogeneous document input data and generate pre-processed documents through format conversion, page segmentation, noise reduction, and optical character recognition, including: Receive multi-source heterogeneous document input data, perform format conversion operations to generate converted documents in a unified image format; Perform page separation on the converted document and output independent page images; Perform interference information elimination operations on independent page images and output optimized page images; Optimize page images through optical character recognition processing and output recognition data containing text content and character position coordinates; A pre-processed document is generated based on the recognition data, and the pre-processed document includes text content, character position set, and distribution information of text and non-text areas within the page.

3. The multimodal document content cross-platform parsing system according to claim 2, characterized in that: Based on the preprocessed documents, three types of features are extracted, including spatial layout features, semantic features, and logical structure features, including: Based on the character position set and text and non-text area distribution information in the preprocessed document, spatial layout features are extracted, and the spatial layout features include text line boundary coordinates, character spacing data, table frame corner coordinates and non-text area positions; based on the text content in the preprocessed document, semantic features are extracted, and the semantic features include entity object identification, syntactic structure relationship and semantic association relationship; based on the segmentation identification, title font size difference, list symbol sequence and table frame topology relationship in the preprocessed document, logical structure features are extracted, and the logical structure features include chapter level number, paragraph affiliation, list nesting level and table cell logical structure.

4. The multimodal document content cross-platform parsing system according to claim 3, characterized in that: Using spatial layout features, the document spatial reference system is constructed based on the statistical values ​​of text line inclination angles and the coordinates of the table frame corners. The connection points of the text lines, the physical boundary corners of the table, and the center points of the paragraph spacing are selected as spatial reference points. The spatial calibration parameters are generated based on the coordinate offsets between the reference points. The calibrated spatial layout features are output, including: Calculate the text line tilt angle statistics based on the text line boundary coordinates and character spacing data in the spatial layout features; Based on the statistical values ​​of the text line tilt angle and the coordinates of the table frame corners, a Cartesian coordinate system with the upper left corner of the page as the origin is established; Generate a set of geometric calibration reference points based on the coordinates of the first and last characters of a text line, the coordinates of the outermost corner points of a table, and the coordinates of the center of the spacing between adjacent paragraphs in the spatial layout features; and generate rotation angle and translation distance parameters based on the coordinate offset of the geometric calibration reference point set in the Cartesian coordinate system. All coordinate data in the spatial layout feature are normalized by the rotation angle and translation distance parameters, and the calibrated spatial layout feature is output.

5. The multimodal document content cross-platform parsing system according to claim 4, characterized in that: The semantic features, logical structure features and calibrated spatial layout features are collaboratively analyzed. By mapping and verifying the semantic content and spatial location, and matching and associating the logical structure and spatial distribution, multimodal correlation features are generated, including: Based on the text line boundary coordinates and non-text area positions in the calibrated spatial layout features, the spatial consistency between the position coordinates of the entity objects in the semantic features and the actual coverage area of ​​the text lines is verified, and a mapping verification relationship between semantic content and spatial position is generated. Based on the table frame corner coordinates and paragraph distribution positions in the calibrated spatial layout features, the spatial distribution range of chapter level numbers and title font size differences in the logical structure features are associated with the spatial topological relationship of the list nesting rules to generate a matching association relationship between the logical structure and the spatial distribution. The mapping verification relationship between semantic content and spatial location and the matching association relationship between logical structure and spatial distribution are integrated to generate multimodal association features including semantic location verification identifiers and logical space topology identifiers.

6. The multimodal document content cross-platform parsing system according to claim 5, characterized in that: The multimodal correlation features are input into the pre-trained visual-linguistic multimodal large model. Through combined semantic understanding and structural analysis, the parent-child hierarchical relationship and semantic type identification of document elements are output, including: The multimodal correlation features are input into the pre-trained visual-linguistic multimodal model, where the following operations are performed: Based on the entity object identification and syntactic structure relationship in the semantic features, combined with the title font size difference and chapter level number in the logical structure features, the semantic type identification and spatial distribution range of the chapter title are determined; Based on the text line boundary coordinates and paragraph distribution positions in the calibrated spatial layout features, the spatial inclusion relationship between the spatial distribution range of the chapter title and the actual coverage area of ​​the paragraph is verified, and the parent-child inclusion relationship at the chapter level is established; Based on the semantic association relationship in the semantic features and the table frame topology relationship in the logical structure features, the semantic association and physical boundary nesting relationship between cells are identified, and the hierarchical structure of the nested table is established; Based on the list symbol sequence rules and indentation position coordinates in the logical structure features, the syntactic structure relationship of the list items in the semantic features is matched to establish the nested hierarchical relationship of the list items; Finally, the semantic type identification and parent-child hierarchical relationship of each document element are output.

7. The multimodal document content cross-platform parsing system according to claim 6, characterized in that: The semantic type identifier includes five types of elements: chapter title, paragraph body, table, list item, and picture. The parent-child hierarchical relationship includes three groups of attributes: parent element identifier, child element identifier, and hierarchical depth.

8. The multimodal document content cross-platform parsing system according to claim 7, characterized in that: Based on the parent-child hierarchical relationship and semantic type identification, a tree-like semantic structure is constructed with chapters as parent units, paragraphs as child units, and nested tables and list items as terminal units. The tree-like semantic structure is then converted into structured data output, including: The semantic type identifier and parent-child hierarchical relationship are used as input sources; when the semantic type identifier is a chapter title, a chapter root node is created according to the parent element identifier in the parent-child hierarchical relationship; when the semantic type identifier is a paragraph body, it is mounted under the corresponding chapter node according to the parent element identifier in the parent-child hierarchical relationship; when the semantic type identifier is a table, cell child nodes are recursively constructed according to the hierarchical depth value in the parent-child hierarchical relationship, and linked to the corresponding chapter or paragraph node; when the semantic type identifier is a list item, based on the hierarchical depth value and parent element identifier in the parent-child hierarchical relationship, a list item node chain is established and mounted to the corresponding paragraph node; when the semantic type identifier is a picture, it is linked to the corresponding paragraph or chapter node according to the parent element identifier in the parent-child hierarchical relationship; finally, a tree-like semantic structure including node type, semantic content, parent element identifier, child element identifier and hierarchical depth is generated, and the tree-like semantic structure is converted into a standard structured data format for output.

9. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the system according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, which, when executed by a processor, implements the system according to any one of claims 1 to 8.

Images