Visual document understanding method and system combining optimized layout and multi-modal information

Abstract

The invention relates to a visual document understanding method and system combining optimized layout and multi-modal information. The method comprises the following steps: calculating the gradient of a to-be-processed document through a pixel compression algorithm, deleting a blank redundant region, and outputting a compressed document image; carrying out OCR (Optical Character Recognition) preprocessing on the text, extracting text and layout information, and enhancing the text quality; an XY cutting algorithm is used, and layout information is optimized through operations such as projection and segmentation; and finally, inputting the text information, the optimized layout information and the compressed document image into a visual large model, fusing multi-modal features, outputting key document information, and completing document understanding. Compared with the prior art, the method has the advantages of high accuracy and the like.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a visual document understanding method and system combining optimized layout and multimodal information. Background Art

[0002] Visual document understanding aims to automatically extract structured information from document images or PDF files with complex layouts. In this task, the semantics of text depends not only on its content but also on its two-dimensional layout structure and rich visual features. In practical applications, large language models can only process the textual and layout features of documents, resulting in low document understanding accuracy. Large-scale visual models, due to input resolution limitations, are unable to capture the detailed semantics of documents, hindering the performance of document understanding methods.

[0003] For example, the invention patent with publication number CN119314189A discloses a generative visual document zero-shot information extraction method based on a large model. It extracts three sets of information features through a document pre-training model and combines and encodes them, outputs multimodal features, builds an adapter to bridge and convert multimodal features and layout enhancement features, realizes the effective fusion of the two, and adjusts the dimensions of multimodal features and layout enhancement features to match the requirements of the large language model. Finally, the mixed features and prompt instructions are input into the large language model, and the understanding ability of the large language model is used to generate structured text. However, the process is complicated and does not take into account the limited input resolution of existing large visual models and the inability to fully receive input document feature information, resulting in the inability to accurately capture document detail feature information. The method has a small scope of application.

[0004] For example, the invention patent with publication number CN115527229A discloses a method and system for extracting key information from document images, which uses an OCR tool to extract text box bounding boxes and text content in document images; however, the document layout information extracted by traditional OCR methods often contains errors, which leads to inaccurate final extraction results.

[0005] In summary, the current visual document understanding methods mainly have the following problems:

[0006] (1) The input resolution of existing large-scale visual models is limited and cannot fully receive the input document feature information, resulting in the inability to accurately capture the document's detailed feature information;

[0007] (2) The document layout information extracted by OCR methods often contains errors, such as inaccurate bounding box alignment and disordered text order, which directly affects the performance of downstream document understanding tasks. Summary of the Invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a visual document understanding method and system that combines optimized layout and multimodal information.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] According to one aspect of the present invention, a method for visual document understanding combining optimized layout and multimodal information is provided, the method comprising the following steps:

[0011] S1. Using a pixel compression algorithm to increase the information density of the document to be processed to output a compressed document image;

[0012] S2, performing OCR preprocessing on the compressed document image to extract text information and layout information of the compressed document;

[0013] S3. Optimize the layout information using an XY cutting algorithm and output the optimized layout information;

[0014] S4. Input the text information, optimized layout information and compressed document image into the visual big model, fuse the text, layout and visual features, output key document information, and thus complete the document understanding.

[0015] As a preferred technical solution, the specific steps of using the pixel compression algorithm to increase the document information density in S1 include:

[0016] S11, using the Sobel operator to calculate the horizontal gradient and the vertical gradient of the document to be processed, and synthesizing the gradient map of the document to be processed by the horizontal gradient and the vertical gradient;

[0017] S12, scanning the gradient map of the document to be processed row by row and column by column to obtain the gradient value of each row and each column;

[0018] S13, if the row gradient value and column gradient value of a certain area are both less than the preset row gradient value and column gradient value thresholds, then the area is determined to be a blank redundant area;

[0019] S14. Delete the blank redundant areas in the document to be processed.

[0020] As a preferred technical solution, the specific formula of the gradient map in S11 is:

[0021]

[0022]

[0023] Among them, I G is the gradient map of the document to be processed; G x is the horizontal gradient of the document to be processed, G y is the vertical gradient of the document to be processed; ID is the document image to be processed; S x is the gradient operator in the horizontal direction; S y is the gradient operator in the vertical direction.

[0024] As a preferred technical solution, the specific formulas for the gradient values ​​of each row and each column in S12 are:

[0025]

[0026] Among them, R i is the gradient value and C of the i-th row; j is the sum of the gradient values ​​of the j-th column of the document.

[0027] As an optimal technical solution, S2 performs OCR preprocessing on compressed documents. After obtaining text information, it also includes the step of text quality enhancement on the text information. Text quality enhancement includes text correction, semantic completion and confidence screening; text correction is to perform spelling checking and grammatical correction on OCR, and generate candidate correction results based on the pre-trained language model; semantic completion is to automatically complete the missing text information in OCR recognition using contextual semantics; confidence screening is to assign a confidence score to each recognized text, and re-OCR recognition of text areas below the preset threshold.

[0028] As a preferred technical solution, the layout information in S3 is a sequence of text bounding boxes. The specific steps of optimizing the layout information using the XY cutting algorithm include:

[0029] S31, performing vertical projection of the X axis and horizontal projection of the Y axis on the text bounding box sequence respectively to obtain a horizontal projection outline and a vertical projection outline;

[0030] S32, searching for valley bottoms of horizontal and vertical projection contours, and alternately performing horizontal splitting or vertical splitting at the valley bottoms until the valley bottoms without projection contours are indivisible, thereby constructing a projection tree;

[0031] S33. Collect the leaf nodes of the projection tree as new layout information.

[0032] As a preferred technical solution, before executing S31, the text bounding box sequence is randomly shifted and enhanced, specifically including: for each bounding box in the text bounding box sequence, a random value is generated based on a uniform distribution, and its coordinates are overall translated according to a preset threshold λ and a shift amount θ, and finally an enhanced text bounding box sequence is output.

[0033] As a preferred technical solution, the specific formula of S31 is:

[0034]

[0035]

[0036] Among them, b i is the i-th bounding box in the document; and are the minimum and maximum coordinate values ​​of the i-th bounding box in the x-axis direction respectively; and are the minimum and maximum coordinate values ​​of the i-th bounding box in the y-axis direction respectively; is the vertical projection function of the i-th bounding box in the x-axis direction; is the horizontal projection function of the i-th bounding box in the y-axis direction; V B is the horizontal projection profile; H B is the vertical projection profile.

[0037] According to another aspect of the present invention, a visual document understanding system combining optimized layout and multimodal information is provided, the system comprising a pixel compression module, an information extraction module, a layout information optimization module, and a document understanding module;

[0038] The pixel compression module is used to improve the information density of the document to be processed by using a pixel compression algorithm to output a compressed document image;

[0039] The information extraction module is used to perform OCR preprocessing on the compressed document image and extract the text information and layout information of the compressed document;

[0040] Layout information optimization is used to optimize layout information using the XY cutting algorithm and output the optimized layout information;

[0041] The document understanding module is used to input text information, optimized layout information and compressed document images into the visual big model, fuse text, layout and visual features, and output key document information to complete document understanding.

[0042] As a preferred technical solution, the system also includes an adaptive learning module, which is used to complete the following steps: first, record the user's correction operations on the output key document information and construct a feedback data set; based on the feedback data, according to the processing accuracy of different types of documents, automatically adjust the threshold parameters of the pixel compression algorithm and the segmentation strategy of the XY cutting algorithm.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. This invention utilizes a pixel compression algorithm to generate a compressed document image. It then feeds text information, optimized layout information, and the compressed document image into a large visual model for multimodal feature fusion. This allows the large model to fully absorb the input document feature information and accurately capture the document's detailed features. The text information carries the document's semantic content, the optimized layout information clarifies the text's structural relationships, and the compressed document image preserves the document's original visual characteristics. By fusing these three types of information, the large visual model can analyze documents from multiple perspectives, including semantics, structure, and vision, improving comprehension accuracy and enhancing the model's generalization capabilities.

[0045] 2. In this invention, when using a pixel compression algorithm to increase document information density, the Sobel operator is used to calculate the horizontal and vertical gradients of the document to be processed, creating a composite gradient map. The gradient map is then scanned row by row and column by column to obtain gradient values ​​for each row and column, thereby accurately determining and deleting blank, redundant areas, thereby increasing the document's effective information density. This operation effectively reduces the data volume of the document image, enabling the document to fit within the limited input resolution of the large visual model without losing critical information. The compressed document image not only preserves detailed feature information but also allows for smooth input into the large visual model, resolving the issue of insufficient document feature information due to resolution limitations.

[0046] 3. The present invention performs OCR preprocessing on compressed documents. After obtaining text information, it also includes the step of enhancing the text quality of the text information, performing text correction, semantic completion and confidence screening on the text information, effectively improving the accuracy of the text information and laying the foundation for accurately extracting layout information.

[0047] 4. The present invention optimizes the layout information using the XY cutting algorithm. First, the text bounding box sequence is randomly shifted and enhanced to increase data diversity and improve the adaptability of the algorithm to different layouts; then, the text bounding box sequence is projected vertically on the X axis and horizontally on the Y axis respectively, and the bottom of the projection contour is searched for alternating segmentation to construct a projection tree, and finally accurate layout information is obtained. By collecting the order of tree leaf nodes, the misaligned bounding boxes are reordered and the layout information is enhanced, while retaining the original shape of the text block. This process can effectively correct problems such as inaccurate bounding box alignment and chaotic text order that occur when OCR extracts layout information. For example, when processing report documents with multi-column layout and irregular text arrangement, the XY cutting algorithm can re-sort the text bounding boxes through precise projection and segmentation operations, accurately align them, and correctly arrange the text order, providing reliable layout information support for downstream document understanding tasks and significantly improving the performance of document understanding.

[0048] 5. The adaptive learning module in this invention records user corrections to key document output information and builds a feedback dataset. Based on this feedback data, it automatically adjusts the threshold parameters of the pixel compression algorithm and the segmentation strategy of the XY Slice algorithm based on the processing accuracy of different document types. Through adaptive adjustments, the system can continuously adapt to the processing needs of different document types, continuously improve document understanding performance, enhance the system's practicality and intelligence, and maintain high processing accuracy and efficiency over long-term use. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the steps of a visual document understanding method combining optimized layout and multimodal information in the present invention;

[0050] Figure 2 This is an example diagram of document image input in the embodiment;

[0051] Figure 3 This is an example diagram of layout information generated by the XY cutting algorithm in the embodiment;

[0052] Figure 4 This is an example diagram of document understanding label determination in the embodiment. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0054] Example 1

[0055] In this embodiment, a visual document understanding method combining optimized layout and multimodal information is applied, characterized in that the method steps are as follows: Figure 1 As shown, specifically including:

[0056] S1. Using a pixel compression algorithm to increase the information density of the document to be processed to output a compressed document image;

[0057] S2, performing OCR preprocessing on the compressed document image to extract text information and layout information of the compressed document;

[0058] S3. Optimize the layout information using an XY cutting algorithm and output the optimized layout information;

[0059] S4. Input the text information, optimized layout information and compressed document image into the visual big model, fuse the text, layout and visual features, output key document information, and thus complete the document understanding.

[0060] In this embodiment, the pixel algorithm is used to increase the document information density in step S1, and the process is as follows:

[0061] S11. First, obtain the horizontal gradient and vertical gradient of the document.

[0062]

[0063] Then get the gradient map of the document.

[0064] I G =max(|G x |,|G y |);

[0065] Among them, I G is the gradient map of the document to be processed; G x is the horizontal gradient of the document to be processed, G y is the vertical gradient of the document to be processed; I D is the document image to be processed; S x is the gradient operator in the horizontal direction; S y is the gradient operator in the vertical direction.

[0066] S12. Scan row by row to obtain the sum of the gradient values ​​of each row of the document; and scan column by column to obtain the sum of the gradient values ​​of each column of the document.

[0067]

[0068]

[0069] Among them, R i is the gradient value and C of the i-th row; j is the sum of the gradient values ​​of the j-th column of the document.

[0070] S13, the rows and columns whose row gradient values ​​and column gradient values ​​are both less than the threshold value form a redundant region;

[0071] S14. Delete the blank redundant area in the document.

[0072] In this embodiment, the XY cutting algorithm generates layout information in step S3, including the following process:

[0073] Random shift enhancement process: For each bounding box in the initial text bounding box sequence, a random value is generated based on a uniform distribution, and the bounding box coordinates are selectively shifted as a whole according to the preset threshold λ and shift amount θ to obtain the enhanced bounding box set;

[0074] Recursive valley cutting process: alternately calculate the projection profile of the current bounding box set in the horizontal and vertical directions, detect the valley area in the projection profile, and recursively split the current bounding box set into subsets according to the valley position until each subset meets the termination condition;

[0075] Sequence rearrangement process: According to the leaf node order formed by the recursive cutting process, the bounding box sequence is reconstructed to obtain the optimized text bounding box sequence.

[0076] The specific process is as follows:

[0077] S31 . Perform a vertical projection of the X axis and a horizontal projection of the Y axis on the bounding box in the document, and obtain a horizontal projection outline and a vertical projection outline of the document.

[0078]

[0079] Among them, b i is the i-th bounding box in the document; and are the minimum and maximum coordinate values ​​of the i-th bounding box in the x-axis direction respectively; and are the minimum and maximum coordinate values ​​of the i-th bounding box in the y-axis direction respectively; is the vertical projection function of the i-th bounding box in the x-axis direction; is the horizontal projection function of the i-th bounding box in the y-axis direction; V B is the horizontal projection profile; H B is the vertical projection profile.

[0080] S32, searching for valley bottoms of horizontal and vertical projection contours, and alternately performing horizontal splitting or vertical splitting at the valley bottoms until the valley bottoms without projection contours are indivisible, thereby constructing a projection tree;

[0081] S33. Collect the leaf nodes of the projection tree as new layout information.

[0082] The information extraction process is as follows:

[0083] The document text information and layout information recognized by OCR are used to construct prompt words to guide the model to output specified information; and the compressed document image is obtained using the pixel compression algorithm; the constructed guide words and the compressed document image are sent to the visual large model together to instruct the model to output key information.

[0084] Example 2

[0085] In this embodiment, a visual document understanding system combining optimized layout and multimodal information is used, and the system includes a pixel compression module, an information extraction module, a layout information optimization module, and a document understanding module;

[0086] The pixel compression module is used to improve the information density of the document to be processed by using a pixel compression algorithm to output a compressed document image;

[0087] The information extraction module is used to perform OCR preprocessing on the compressed document image and extract the text information and layout information of the compressed document;

[0088] Layout information optimization is used to optimize layout information using the XY cutting algorithm and output the optimized layout information;

[0089] The document understanding module is used to input text information, optimized layout information and compressed document images into the visual big model, fuse text, layout and visual features, and output key document information to complete document understanding.

[0090] The system also includes an adaptive learning module, which is used to complete the following steps: first, record the user's correction operations on the output key document information and build a feedback data set; based on the feedback data, according to the processing accuracy of different types of documents, automatically adjust the threshold parameters of the pixel compression algorithm and the segmentation strategy of the XY cutting algorithm.

[0091] The adaptive learning module builds a feedback dataset by recording user corrections to key output document information. Based on this feedback data, it can automatically adjust the threshold parameters of the pixel compression algorithm and the segmentation strategy of the XY cut algorithm according to the processing accuracy of different document types. In actual applications, different types of documents, such as contracts, reports, and invoices, have different formats and content characteristics. The adaptive learning module can optimize algorithm parameters to address these differences. For contract documents with more standardized formats and high text density, the threshold of the pixel compression algorithm may need to be adjusted to more accurately retain valid information. For report documents with complex layouts and numerous charts, the segmentation strategy of the XY cut algorithm can be optimized to better handle complex layout structures. Through this adaptive adjustment, the system can continuously adapt to the processing needs of different document types, continuously improve document comprehension performance, enhance the system's practicality and intelligence, and maintain high processing accuracy and efficiency over long-term use.

[0092] The system workflow is as follows. The input document is Figure 2 For example, the system's working goal is to determine the category to which a document text block belongs.

[0093] First enter Figure 2The visual document shown in the figure is then compressed using a pixel compression algorithm to remove redundant blank areas and increase the document's effective information density. OCR preprocessing is then performed to obtain the visual document's text information and layout information (the bounding box corresponding to the document text). The layout information is then regenerated using an XY cutting algorithm.

[0094] Utilize pixel compression algorithms to compress documents, remove blank and redundant areas, and increase the effective information density of documents. Specific examples include:

[0095] First, the Sobel operator is used to calculate the horizontal and vertical gradients of the document to be processed, and then a gradient map is synthesized to accurately capture the edge information of the document image. Next, the gradient values ​​of each row and column of the document image are obtained by scanning row by row and column by column; based on these gradient values, if the row gradient value and column gradient value of a certain area are both less than the preset threshold, the area can be determined to be a blank redundant area and deleted. This series of operations can effectively increase the information density of the document, remove unnecessary blank parts, reduce the amount of data for subsequent processing, and improve overall processing efficiency. For example, when processing a large number of scanned documents, blank areas may occupy a large storage space. This algorithm can significantly reduce storage requirements. At the same time, in subsequent OCR recognition and other links, due to the reduction in data volume, computing resource consumption is reduced, and the recognition speed is accelerated.

[0096] OCR preprocessing of compressed document images extracts text and layout information. Subsequent text quality enhancement is crucial. Text correction uses a pretrained language model to perform spelling checks and grammatical corrections on the OCR results, generating candidate corrections. This helps correct text errors caused by character recognition errors or formatting issues during the OCR process, improving text accuracy. For example, character confusion is common in OCR recognition of scanned handwritten documents, and text correction effectively addresses this issue. Semantic completion automatically fills in missing text information during OCR recognition using contextual semantics. Documents may contain blurred text or failed recognition. Semantic completion, based on logical relationships between the preceding and following text, infers and supplements the missing content, making the text more complete and coherent. Confidence filtering assigns a confidence score to each recognized text, and re-OCR is performed on text areas that fall below a preset threshold. This ensures high-quality final output, reduces errors caused by inaccurate recognition, and provides a reliable data foundation for subsequent document understanding.

[0097] Layout information is crucial for document understanding, and the XY cut algorithm can effectively optimize this information. The XY cut algorithm regenerates layout information specifically including:

[0098] First, the document bounding box is projected horizontally and vertically; then the valleys of the horizontal and vertical projection contours are searched and horizontal and vertical cutting is performed alternately until there is no valley for each node to be split; finally, the nodes are collected as new layout information. The XY cutting algorithm generates layout information as follows: Figure 3 This approach can untangle complex text layouts within documents and clearly present the hierarchical relationships between text. When processing documents with complex structures such as multiple columns, titles, and body text, the XY Slice algorithm accurately demarcates different areas, clarifying the document's logical structure. This facilitates a better understanding of document content when subsequently integrating it with text information and visual features, improving the accuracy and efficiency of document comprehension.

[0099] Finally, the text information, optimized layout information and compressed document image are input into the visual model to fuse the text, layout and visual features, and output the key document information, thus completing the document understanding. Figure 4 As shown in the figure, multimodal information fusion, combining text information, optimized layout information, and compressed document images into a large visual model, effectively achieves document understanding. Text information provides the semantic content of the document, optimized layout information clarifies the structural relationships of the text, and compressed document images preserve the original visual features of the document. The large visual model possesses powerful feature learning and processing capabilities, enabling comprehensive analysis of this multimodal information. Compared to traditional methods that rely solely on single modal information, multimodal fusion provides a more comprehensive understanding of documents. When processing documents containing mixed text and images, it not only recognizes the text content but also understands the relationship between images and text based on layout information, combining image features to more accurately interpret the overall meaning of the document. In terms of performance metrics, in several document understanding benchmarks, multimodal information fusion methods have shown significant improvements in precision and recall compared to single-modal methods. This approach can better meet the needs of document understanding in practical applications, playing a vital role in scenarios such as intelligent document review and document information extraction.

[0100] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A visual document understanding method combining optimized layout and multimodal information, characterized in that: The method steps include: S1. Using a pixel compression algorithm to increase the information density of the document to be processed to output a compressed document image; S2, performing OCR preprocessing on the compressed document image to extract text information and layout information of the compressed document; S3. Optimize the layout information using an XY cutting algorithm and output the optimized layout information; S4. Input the text information, optimized layout information and compressed document image into the visual macro model, perform text, layout and visual feature fusion, output key document information, and thus complete document understanding.

2. A visual document understanding method combining optimized layout and multimodal information according to claim 1, characterized in that: The specific steps of using the pixel compression algorithm to increase the document information density in S1 include: S11, using the Sobel operator to calculate the horizontal gradient and the vertical gradient of the document to be processed, and synthesizing the gradient map of the document to be processed by the horizontal gradient and the vertical gradient; S12, scanning the gradient map of the document to be processed row by row and column by column to obtain the gradient value of each row and each column; S13, if the row gradient value and column gradient value of a certain area are both less than the preset row gradient value and column gradient value thresholds, then the area is determined to be a blank redundant area; S14. Delete the blank redundant areas in the document to be processed.

3. The method for visual document understanding combining optimized layout and multimodal information according to claim 2, characterized in that: The specific formula of the gradient map in S11 is: I G =max(|G x |,|G y |); G x =I D *S x ; G y =I D *S y ; Among them, I G is the gradient map of the document to be processed; G x is the horizontal gradient of the document to be processed, G y is the vertical gradient of the document to be processed; I D is the document image to be processed; S x is the gradient operator in the horizontal direction; S y is the gradient operator in the vertical direction.

4. The method for visual document understanding combining optimized layout and multimodal information according to claim 3, characterized in that: The specific formulas for the gradient values ​​of each row and each column in S12 are: Among them, R i is the gradient value and C of the i-th row; j is the sum of the gradient values ​​of the j-th column of the document.

5. The method for visual document understanding combining optimized layout and multimodal information according to claim 1, characterized in that: The S2 performs OCR preprocessing on the compressed document, and after obtaining the text information, it also includes the step of enhancing the text quality of the text information, wherein the text quality enhancement includes text correction, semantic completion and confidence screening; the text correction is to perform spelling checking and grammatical correction on the OCR, and generate candidate correction results based on the pre-trained language model; the semantic completion is to automatically complete the missing text information in the OCR recognition using contextual semantics; the confidence screening is to assign a confidence score to each recognized text, and re-OCR recognition is performed on the text area below the preset threshold.

6. The method for visual document understanding combining optimized layout and multimodal information according to claim 1, characterized in that: The layout information in S3 is a sequence of text bounding boxes. The specific steps of optimizing the layout information using the XY cutting algorithm include: S31, performing vertical projection of the X axis and horizontal projection of the Y axis on the text bounding box sequence respectively to obtain a horizontal projection outline and a vertical projection outline; S32, searching for valley bottoms of horizontal and vertical projection contours, and alternately performing horizontal splitting or vertical splitting at the valley bottoms until the valley bottoms without projection contours are indivisible, thereby constructing a projection tree; S33. Collect the leaf nodes of the projection tree as new layout information.

7. The method for visual document understanding combining optimized layout and multimodal information according to claim 6, characterized in that: Before executing S31, the text bounding box sequence is randomly shifted and enhanced, specifically including: for each bounding box in the text bounding box sequence, generating a random value based on a uniform distribution, and performing an overall translation on its coordinates according to a preset threshold λ and a shift amount θ, and finally outputting an enhanced text bounding box sequence.

8. The method for visual document understanding combining optimized layout and multimodal information according to claim 6, characterized in that: The specific formula of S31 is: Among them, b i is the i-th bounding box in the document; and are the minimum and maximum coordinate values ​​of the i-th bounding box in the x-axis direction respectively; and are the minimum and maximum coordinate values ​​of the i-th bounding box in the y-axis direction respectively; is the vertical projection function of the i-th bounding box in the x-axis direction; is the horizontal projection function of the i-th bounding box in the y-axis direction; V B is the horizontal projection profile; H B is the vertical projection profile.

9. A visual document understanding system combining optimized layout and multimodal information, characterized in that: The system applies a visual document understanding method combining optimized layout and multimodal information as described in any one of claims 1 to 8, and the system includes a pixel compression module, an information extraction module, a layout information optimization module, and a document understanding module; The pixel compression module is used to increase the information density of the document to be processed by using a pixel compression algorithm to output a compressed document image; The information extraction module is used to perform OCR preprocessing on the compressed document image to extract the text information and layout information of the compressed document; The layout information optimization is used to optimize the layout information using an XY cutting algorithm and output the optimized layout information; The document understanding module is used to input the text information, optimized layout information and compressed document image into the visual macro model, perform text, layout and visual feature fusion, and output key document information, thereby completing document understanding.

10. A visual document understanding system combining optimized layout and multimodal information according to claim 9, characterized in that: The system also includes an adaptive learning module, which is used to complete the following steps: first, record the user's correction operations on the output key document information and build a feedback data set; based on the feedback data, according to the processing accuracy of different types of documents, automatically adjust the threshold parameters of the pixel compression algorithm and the segmentation strategy of the XY cutting algorithm.

Images