A multi-level structure information inference method and device for complex document images

Abstract

The application discloses a multi-level structure information inference method and device for a complex document image, and relates to the technical field of complex document image processing. The method comprises the following steps: performing page range division by using a minimum covering rectangular frame calculation method according to a first set of page elements, and obtaining page structure information; performing column range division by using a connected subgraph solving method according to the page structure information based on geometric expansion-intersection testing, and obtaining column structure information; performing same-element merging by using an element alignment method according to the column structure information, and obtaining line structure information; performing covering element splitting by using a center point calculation method according to the line structure information, and obtaining line-internal structure information; and recognizing the element relationship between lines and columns, and constructing multi-level structure information of the complex document image. The application is a real-time and efficient multi-level result information inference method for a complex document image.

Description

Technical Field

[0001] This invention relates to the field of complex document image processing technology, and in particular to a method and apparatus for inferring multi-level structural information of complex document images. Background Technology

[0002] Multi-level structural information inference of complex document images refers to a technical system that integrates visual, semantic and geometric cues to perform multi-granular hierarchical modeling and joint reasoning of information contained in a document in scenarios with highly flexible layout, dense content and diverse element types.

[0003] Currently, in the field of complex document image detection and recognition, the main methods involve using models and algorithms such as optical character recognition, page layout analysis, mathematical formula detection, and reading order recognition, combined with specific domain rules, to extract text and images. The main problem is that visually rich document images are characterized by diverse elements and complex layout relationships. Furthermore, a large number of documents often need to be processed in a short time. When processing complex document images, single model algorithms suffer from limitations such as a limited number of detectable elements, difficulty in simultaneously detecting and analyzing elements at multiple levels, and low real-time speed in inferring relationships between elements. This results in high costs and poor economic efficiency for inferring structural information from complex document images, making it difficult to meet the needs of users in real-world scenarios who can freely extract the position and relationships of elements such as the page margins and columns. This limits the effectiveness of applications such as high-precision recovery of rich text information from complex document images and evaluation of page layout quality and style characteristics.

[0004] The purpose of this invention is to correctly determine the sequential and containment relationships between elements at different levels in a document image with a complex layout by combining hierarchical modeling and rule-driven approaches, based on existing layout analysis, text detection, and formula detection models. This enables the correct inference of multi-level structural information in complex documents, providing a basis for downstream tasks such as text information extraction and significantly simplifying the complexity of downstream tasks.

[0005] In the existing technology, there is a lack of a real-time and efficient method for inferring multi-level result information for complex document images. Summary of the Invention

[0006] To address the technical problems of existing technologies when processing complex document images, such as limited types of detectable elements, difficulty in simultaneously detecting and recognizing elements at multiple levels, and high computational requirements for inferring relationships between elements, this invention provides a method and apparatus for inferring multi-level structural information from complex document images. The technical solution is as follows:

[0007] On the one hand, a method for inferring multi-level structural information of complex document images is provided. This method is implemented by a multi-level structural information inference device and includes:

[0008] Acquire complex document images; perform layout analysis on the complex document images to obtain layout analysis results; perform data analysis and processing on the layout analysis results to obtain the first set of page elements;

[0009] Based on the first set of elements, the page range is divided using the minimum covering rectangle calculation method to obtain page structure information;

[0010] Based on the geometric expansion-intersection test, the column range is divided according to the page structure information using the connected subgraph solution method to obtain the column structure information;

[0011] Based on the column structure information, use the element alignment method to merge similar elements to obtain the line structure information;

[0012] Based on the row structure information, the center point calculation method is used to split the covering elements to obtain the in-row structure information;

[0013] Information optimization is performed based on page structure information, column structure information, line structure information, and inline structure information. The element relationships between lines and columns are identified to construct multi-level structural information for complex document images.

[0014] On the other hand, a multi-level structural information inference apparatus for complex document images is provided. This apparatus is applied to a multi-level structural information inference method for complex document images, and the apparatus includes:

[0015] The layout element acquisition module is used to acquire complex document images; it performs data analysis and processing on the complex document images to obtain the first set of layout elements.

[0016] The page information inference module is used to acquire complex document images; perform layout analysis on the complex document images to obtain layout analysis results; and perform data analysis and processing on the layout analysis results to obtain the first set of page elements.

[0017] The column information inference module is used to divide the column range based on the geometric expansion-intersection test and the page structure information using the connected subgraph solution method to obtain the column structure information.

[0018] The line break information inference module is used to merge similar elements based on the column structure information using element alignment methods to obtain the line break structure information.

[0019] The inline information inference module is used to split the covering elements based on the line structure information and the center point calculation method to obtain the inline structure information.

[0020] The multi-level information inference module is used to optimize information based on page structure information, column structure information, line structure information and inline structure information, and to identify the element relationships between lines and columns to construct multi-level structural information of complex document images.

[0021] On the other hand, a multi-level structural information inference device is provided, the multi-level structural information inference device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for multi-level structural information inference of complex document images.

[0022] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for inferring multi-level structural information of complex document images.

[0023] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0024] This invention proposes a method for inferring multi-level structural information from complex document images. Based on existing layout analysis, text detection, and formula detection models, it infers structural information at four levels—full-page, column, inline, and inline—from document images with complex layouts. It correctly determines the order, inclusion, and merging relationships between elements at each level, achieving accurate inference of multi-level structural information in complex documents. This provides a basis for downstream tasks such as text information extraction, significantly simplifying the complexity of downstream tasks. This invention can recursively parse structural information at four levels: full-page, column, block, and inline, and quickly infer the order and inclusion relationships of each element, improving the automated processing level of visually rich documents such as technical documents and academic papers. This invention is a real-time and efficient method for inferring multi-level result information from complex document images. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a method for inferring multi-level structural information of complex document images provided by an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of a complex document image element layout provided by an embodiment of the present invention;

[0028] Figure 3 This is a block diagram of a multi-level structural information inference device for complex document images provided in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the structure of a multi-level structural information inference device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0031] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0032] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0033] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0034] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0035] This invention provides a method for inferring multi-level structural information from complex document images. This method can be implemented using a multi-level structural information inference device, which can be a terminal or a server. Figure 1 The flowchart shown is for a method to infer multi-level structural information from complex document images. The processing flow of this method may include the following steps:

[0036] S1. Obtain a complex document image; perform layout analysis on the complex document image to obtain the layout analysis results; perform data analysis and processing on the layout analysis results to obtain the first set of layout elements.

[0037] In one feasible implementation, the core of analyzing and understanding complex document images lies in decomposing the visual structure, semantic content, and logical relationships within the image into a set of computable and recombinable structured elements. These elements can include text, formulas, seals, etc., and each element includes information such as its corresponding image, positional coordinates, and element type. The results of the layout analysis primarily include the detection results from publicly available model algorithms for optical character recognition, layout analysis, and mathematical formula detection.

[0038] S2. Based on the first set of elements, use the minimum covering rectangle calculation method to divide the page range and obtain page structure information.

[0039] Optionally, based on the first set of elements, the page area is divided using the minimum covering rectangle calculation method to obtain page structure information, including:

[0040] Filter the non-main elements from the first set of elements to obtain the second set of elements.

[0041] The minimum covering rectangle is calculated based on the element set of the second page to obtain the element set of the first page.

[0042] Based on the first set of elements, the scope is divided according to the second set of elements to obtain the set of header elements, the set of footer elements, the set of left footnote elements, and the set of right footnote elements.

[0043] Based on the first set of heart element, header element, footer element, left footnote element, and right footnote element, obtain the page structure information.

[0044] In one feasible implementation, the first set of page elements is read in. Each of them All contain the coordinate range information of the element (containing The four values ​​represent the x-coordinate of the top-left corner, the x-coordinate of the top-right corner, the y-coordinate of the top-left corner, and the y-coordinate of the top-right corner in the image, respectively. The origin of the document image coordinate system is the top-left corner of the image, with the positive direction of the x-coordinate to the right and the positive direction of the y-coordinate downwards. The coordinate conventions thereafter are the same.

[0045] like If the type is header, footer, margin, or stamp, then remove it from the collection. Remove from the middle to obtain the second set of elements. The width of the image in the current document is defined as follows: The height is .

[0046] against For each element in the text, select the smallest bounding box containing all elements as the text area. The coordinate range is determined and added along with its type to the final element corresponding to the current document image. If there is no page margin, the current document image will be considered a page without a page margin. The page margin... The formula for calculating the coordinate position is as follows (1):

[0047] (1);

[0048] Assume the coordinate range of the text area in the document image is: The area above the text area is then marked as the header area, meaning the footer area is... The area located below the text area is designated as the footer area; the footer area is defined as follows: The area located on the left side of the text area is marked as the left footnote area, that is, the range of the left footnote area is... The area located on the right side of the text area is marked as the right footnote area, that is, the range of the right footnote area is... The left and right side note areas are collectively referred to as the side note area.

[0049] S3. Based on the geometric expansion-intersection test, the column range is divided using the connected subgraph solution method according to the page structure information to obtain the column structure information.

[0050] Optionally, based on the geometric expansion-intersection test, and according to the page structure information, a connected subgraph solution method is used to divide the column range and obtain the column structure information, including:

[0051] The first set of core elements in the page structure information is combined with fine element filtering to obtain the second set of core elements.

[0052] Based on the second set of core elements, the geometric expansion-intersection test method is used to determine whether the core elements can be passed through, and the first set of core elements that can be passed through and the first set of core elements that cannot be passed through are obtained.

[0053] An undirected graph is constructed based on the first set of elements that can be displayed across the entire screen and the first set of elements that cannot be displayed across the entire screen, resulting in an undirected graph that can be displayed across the entire screen and an undirected graph that cannot be displayed across the entire screen. Based on the undirected graph that can be displayed across the entire screen and the undirected graph that cannot be displayed across the entire screen, the region is divided using the connected subgraph solution method, resulting in a second set of elements that can be displayed across the entire screen and a second set of elements that cannot be displayed across the entire screen.

[0054] Based on the horizontal overlap rule of elements, an undirected graph is constructed according to the second set of non-full-width elements to obtain a non-full-width filtering undirected graph; based on the non-full-width filtering undirected graph, the region is divided using the connected subgraph solution method to obtain a set of non-full-width single-column elements.

[0055] Based on the second set of elements that can be displayed across the entire column and the set of elements that cannot be displayed across the entire column, an undirected adjacency graph is constructed using a sorting and grouping method to obtain a column-based undirected adjacency graph. Based on the column-based undirected adjacency graph, a secondary column division is performed using a connected subgraph solving method to obtain the second set of elements that can be displayed across the entire column and the second set of elements that cannot be displayed across the entire column. Based on the second set of elements that can be displayed across the entire column and the second set of elements that cannot be displayed across the entire column, the column structure information is obtained.

[0056] In one feasible implementation, elements that are too narrow in width are filtered out within the text area to avoid interference from small elements in the subsequent inference of column information.

[0057] In the page layout analysis results, less than Remove 1 / 10 of it. That is, if Then remove the element from the set. Remove the middle part to get the set of elements. The following will be done for each one in The elements in the text are used to determine whether they can be displayed across columns.

[0058] For each , building and The set of elements that do not intersect with each other That is, each They are all sets Zhongyu A set of elements that do not overlap. Elements are arranged one by one according to their position in the center of the page. The boundary was changed The boundary coordinates form the element The process is as follows (2):

[0059] (2);

[0060] Calculate the elements after the range has been changed. With sets The intersection of each element. If If it does not intersect with any element, then mark it. Elements that can span the entire width of the page are allowed; otherwise, they are marked as such. This refers to elements that are not full-width elements.

[0061] Based on each element The location and spanning capability markers, through each element The intersection relationship between the elements is used to construct and generate both full-width and non-full-width areas within the area of ​​the page center.

[0062] Will The elements in are arranged according to The coordinates (i.e., the ordinates of the element centers) are arranged in ascending order. Construct an undirected graph. Used for judgment The adjacency relationship between them. Among them, Each element in is Find a vertex in the graph. Construct the graph according to the following rules. Each edge in: If and Adjacent (i.e.) )and and If all elements are span-width elements, then connect the edges. and ;like and Adjacent (i.e.) )and and If all elements are not full-width elements, then connect the edges. and After obtaining the connectivity relationships, the connected subgraph construction algorithm in graph theory is used to find the... Each of the connected subgraphs In each connected subgraph, calculate the minimum rectangular boundary range corresponding to all elements within each connected subgraph. As the boundary range of each connected subgraph, each The coordinate parameters are as follows (3):

[0063] (3);

[0064] like If every element inside is a full-width element, then... Mark it as a full-width area. If If each element inside is a non-full-width element, then... Mark it as a non-full-width area.

[0065] The initial division and segmentation of the full-width and non-full-width areas yields the initial column layout. The resulting set of areas... Each element is associated separately based on whether it can span the entire width of the page. Each page element is processed and merged to obtain the initial column layout. An empty set is constructed beforehand for this purpose. ,Towards Add elements to the collection, and after the process is complete, the collection will be... This is the result of the initial column division.

[0066] if If it can be displayed across the entire column, then directly... Add to collection In the middle, this allows the area that can be spanned to be separated into a single column, if For non-full-width columns, calculate the relationship as follows: Related columns, and add each column separately to middle.

[0067] For each non-full-width area Test one by one Each page element To determine whether it falls on Inside. Within, elements Falling in the area The criteria for determining the content are as follows: center point coordinates Falling in the non-full-length area internal.

[0068] Once the judgment is complete, it will fall in the non-full-width area. All inside Collect them to form a set .by For each element vertex in the graph, construct an undirected graph. .

[0069] According to the rules Add layout elements that meet certain relationships as images The edges in the middle. Among them, if two different page elements... and There is overlap in the horizontal direction (i.e.) or ), then the edge Add to graph .

[0070] Using the algorithm for finding connected subgraphs of an undirected graph in graph theory, find... Internal connected subgraphs For each connected subgraph Calculate the minimum rectangular boundary range corresponding to all elements within each connected subgraph to obtain the rectangular bounding box of each connected subgraph. At this point, each rectangular bounding box is a preliminarily inferred single-column region. Each bounding box is then added as a single-column region to... In this way, the initial column division results are obtained.

[0071] In document images with complex layouts, there is a problem where single-column areas are fragmented, and isolated non-full-width areas may be identified as single-column areas. Therefore, the method described in this step is needed to perform secondary column division.

[0072] Secondary column division, based on the obtained full-width areas and the initial column division result, merges adjacent full-width areas and non-full-width sub-columns to form a column division result that better conforms to the layout rules. The specific steps are as follows:

[0073] Obtain the result set formed by the initial column division Assuming The CCP Individual columns. For For each sub-column, calculate the y-coordinate of the center point of its rectangular area (i.e., the average of the minimum and maximum y-coordinates in the document image). Then, arrange them in ascending order of their center point y-coordinates. For sub-columns with the same y-coordinate, arrange them in ascending order of their x-coordinates. Each sorted sub-column is denoted as follows: .

[0074] For sorted sub-columns, based on their spanning property (either spanning or not spanning), adjacent sub-columns with the same property are grouped together, creating an alternating effect of spanning groups and non-spanning groups. First, groups are numbered sequentially starting from 1. Then, within each group, each sub-column is numbered sequentially starting from 1, ensuring that each sub-column is assigned a unique two-dimensional array. serial number.

[0075] By traversing the graph Establish adjacency relationships between each element at each vertex, as... The edges in the text. When establishing adjacency relationships, first assign a column number to each sub-column, and then establish adjacency relationships based on the column number. There are three possible scenarios: ① If the sequence number is... If a sub-column is a non-full-width area, then iterate through the groups containing that sub-column, judging them by their x-coordinates in ascending order. If two sub-columns within a group have intersecting x-coordinates, they are considered to be in the same column, and column numbers are assigned starting from zero, with an adjacency edge established between the two sub-columns. If adjacent sub-columns do not intersect, increment the column number of the next sub-column by 1. ② For two adjacent groups, if the previous group... This is a full-width area; the next group... If it belongs to a non-full-width area, then first... Find the last column inside, then... Find the first element that intersects with the horizontal coordinate of this column, and establish an adjacent edge between these two sub-columns. ③ For two adjacent groups, if the previous group... This is a non-full-width area; the next group. If it belongs to a full-width area, then first... Find the first column inside, then in Find the column that intersects with the horizontal coordinate of this column and has the largest sequence number. If a sub-column only intersects with this sub-column, then an adjacent edge is created between the two sub-columns, and the result is recorded. This column The number. Next, from... Take the second sub-column and compare it with the column. If the x-coordinates intersect, then in The second sub-column and column Establish adjacent edges between the two groups. If they do not intersect, terminate the determination of the adjacency relationship between the two groups.

[0076] Compute using the connected subgraph algorithm for undirected graphs from graph theory. Each connected subgraph, according to The minimum rectangular coverage of each column in the image, including the sub-images, is calculated and used as the result of the secondary column division. This is then labeled as the sub-column in the current document image.

[0077] S4. Based on the column structure information, use the element alignment method to merge similar elements to obtain the line structure information.

[0078] Optionally, based on the column structure information, element alignment methods are used to merge similar elements to obtain line break structure information, including:

[0079] Based on the column structure information, the column width alignment method is used to merge similar elements at the paragraph level to obtain a collection of various types of paragraph blocks;

[0080] Based on the collection of various types of paragraph blocks, the same type of elements at the row level are merged using the column height alignment method to obtain a collection of various types of row boxes;

[0081] The collection of various types of line boxes is determined as the line structure information.

[0082] One feasible implementation involves associating layout elements with corresponding sub-columns and calculating the expanded area of ​​each layout element within the sub-column. That is, for each layout element... Set the starting point of its horizontal coordinate to the starting point of the horizontal coordinate of the sub-column, and set the ending point of its horizontal coordinate to the ending point of the horizontal coordinate of the sub-column. This ensures that each... All expanded into a rectangular area .

[0083] Compare the overlap of each rectangular region, merge overlapping elements to obtain each paragraph. If the expanded region... and The length of overlap in the longitudinal direction exceeds or If the height is 0.2 times, the two regions will be merged into one region.

[0084] Each merged region is treated as a block, and its location and type are labeled based on the types of elements it contains. Specifically, a block containing hierarchical headings is a heading block; a block containing inline formulas without numbering is an "unnumbered inline formula"; and a block containing both inline formulas and numbering is a "numbered inline formula". A block containing only paragraph text is a plain text block. A block containing both paragraph text and mathematical formulas is a text block containing formulas. A block containing tables is a table block; and a block containing images is an illustration block. Labeling these types of information allows for more effective content identification for different types of blocks.

[0085] Based on the results of text detection and formula detection, repeating the above steps determines whether text and formulas are located at the same vertical height. Elements at the same vertical height are merged to obtain the position range and type information of each row within the sub-column. The smallest rectangle covering elements at the same vertical height defines the area range of that row. If the row does not contain inline formulas, it is a text row; if it contains inline formulas, it is a text row containing formulas. Labeling this type information facilitates subsequent use of different optical character recognition algorithms to identify the content of different rows.

[0086] S5. Based on the line structure information, use the center point calculation method to split the covering elements and obtain the in-line structure information.

[0087] Optionally, based on the line structure information, the covering elements are split using a center point calculation method to obtain the in-line structure information, including:

[0088] Based on the line structure information, obtain the set of text boxes and the set of formula boxes;

[0089] The center point is calculated based on the set of formula boxes to obtain the set of center points of the formula boxes;

[0090] Based on the set of center points of the formula box, the set of text boxes is split and validated to obtain the set of text boxes to be split and the set of text boxes not to be split.

[0091] Based on the set of text boxes to be split, and according to the set of formula boxes, select the formulas covered in the text boxes to obtain the set of covered formula boxes;

[0092] Based on the set of covered formula boxes, the text boxes to be split are split using the ascending horizontal coordinate sorting method to obtain the set of split text boxes;

[0093] Based on the split text box set, perform a merge check on the text boxes that are not split, and partially merge the text boxes of the same type to obtain the merged text box set and the unmerged text box set;

[0094] Obtain inline structure information based on the merged text box set and the unmerged text box set.

[0095] In one feasible implementation, the need to split the text region is determined based on whether the text region and the formula region overlap. Specifically, each group of text and formula regions located on the same line is selected; if the center point of the rectangle corresponding to the formula region falls within a certain text region, then that text region needs further subdivision.

[0096] For a text region that needs to be split, it is divided into two parts from the formula region it covers. One part is located on the left side of the formula region, and its maximum horizontal coordinate is the minimum horizontal coordinate of the formula region. The other part is located on the right side of the formula region, and its minimum horizontal coordinate is the maximum horizontal coordinate of the formula region.

[0097] S6. Optimize information based on page structure information, column structure information, line structure information and inline structure information, and identify the element relationships between lines and columns to construct multi-level structural information of complex document images.

[0098] Optionally, information optimization is performed based on page structure information, column structure information, line break structure information, and inline structure information, and the element relationships between lines and columns are identified to construct multi-level structural information for complex document images, including:

[0099] Based on the inline structure information, the minimum covering rectangle calculation method is used to optimize the structure information according to the line structure information to obtain the first optimized line structure information.

[0100] Based on the optimized row structure information, the minimum covering rectangle calculation method is used to optimize the structure information according to the column structure information to obtain the first optimized column structure information.

[0101] Based on the optimized column structure information, the minimum covering rectangle calculation method is used to optimize the structure information according to the page structure information to obtain the first optimized page structure information.

[0102] Based on a one-dimensional horizontal sorting rule, the in-row structural information is sorted by in-row elements to obtain the first optimized in-row structural information.

[0103] Based on the one-dimensional vertical sorting rule and the two-dimensional single-column sorting rule, the elements of the first optimized line structure information are sorted between rows to obtain the second optimized line structure information.

[0104] Based on the one-dimensional vertical sorting rule, the elements between columns of the first optimized column structure information are sorted to obtain the second optimized column structure information.

[0105] Based on the page layout order rules, the page elements of the first optimized page structure information are sorted to obtain the second optimized page structure information;

[0106] Based on the second optimized row structure information and the second optimized column structure information, the adjacent element type retrieval method is used to identify the merging relationship and obtain the element relationship information.

[0107] Based on the second optimized page structure information, the second optimized column structure information, the second optimized line structure information, the first optimized inline structure information, and the element relationship information, the multi-level structure information of the document image is obtained.

[0108] In one feasible implementation, the present invention divides the element sorting rules in complex document images into three categories. Subsequent element sorting at each level uses a combination of one or more of these methods. The three categories are: ① One-dimensional horizontal sorting rules, i.e., given a column of elements... According to the leftmost horizontal coordinate of each element Or the x-coordinate of the center point ① Sort and number the elements in ascending order; ② One-dimensional vertical sorting rule, that is, given a column of elements According to the vertical top coordinate of each element Or the ordinate of the center point ③ Sort and number elements in ascending order; ④ Two-dimensional single-column sorting rules. Given a column of elements... Calculate whether two elements overlap vertically (the length of the vertical overlap exceeds 0.2 times the height of one of the elements), treat the overlapping elements as a row, sort the row according to the one-dimensional vertical sorting rule, and then sort the elements in the same row according to the one-dimensional horizontal sorting rule to obtain the global order of the elements.

[0109] Based on the coverage of elements at different levels, the containment relationships between elements at the inline, column, page, and global levels are determined. Specifically, for elements at a certain level... If the coordinates of the center point of the element below it are inside this element, then this element is determined. Includes the corresponding child elements. Adjusts the boundary of this element to the smallest rectangle that covers all its child elements.

[0110] According to the sorting rules, elements at the inline, column, page, and global levels are sorted in order of priority. For inline text and formula areas, the order of elements is determined according to a one-dimensional horizontal sorting rule.

[0111] For inline and block-level elements within the same column, the order of elements is determined according to one-dimensional vertical sorting rules, and the order of inline text and formula areas within the same column is determined according to two-dimensional single-column sorting rules and inclusion relationships.

[0112] For each column, the order of each sub-column is determined according to a one-dimensional vertical sorting rule. Based on the order of the sub-columns, the order of elements at various levels within the text area, such as inline, interline, and block elements, is determined according to their inclusion relationships. The reading order of all elements within the document image is determined according to the order of header, left footnote, text area, right footnote, and footer.

[0113] Based on the order in which they were read, infer the merging relationships between elements in adjacent rows and adjacent sub-columns. Specifically, for two adjacent rows within the same sub-column, if the sequence number is... The last element of the row is text or a formula, and its sequence number is... The type and index of the first element of the row are If the last element of a row is the same, then the two elements are determined to have a merge relationship. If the last element of an adjacent sub-column is text or a formula, and the first element of the next sub-column is also text or a formula, then the two elements are also determined to have a merge relationship. By establishing a merge relationship, the images of two parts can be stitched together for recognition when identifying the content of an element, or the images of two parts can be merged into a single line of text or a single formula after content recognition.

[0114] This invention proposes a method for inferring multi-level structural information from complex document images. Based on existing layout analysis, text detection, and formula detection models, it infers structural information at four levels—full-page, column, inline, and inline—from document images with complex layouts. It correctly determines the order, inclusion, and merging relationships between elements at each level, achieving accurate inference of multi-level structural information in complex documents. This provides a basis for downstream tasks such as text information extraction, significantly simplifying the complexity of downstream tasks. This invention can recursively parse structural information at four levels: full-page, column, block, and inline, and quickly infer the order and inclusion relationships of each element, improving the automated processing level of visually rich documents such as technical documents and academic papers. This invention is a real-time and efficient method for inferring multi-level result information from complex document images.

[0115] Figure 3 This is a block diagram of a multi-level structural information inference device for complex document images provided in an embodiment of the present invention. This device is used for a method of inferring multi-level structural information from complex document images. (Refer to...) Figure 3The device includes a first acquisition module 310, a second acquisition module 320, an inspection module 330, and a training module 340. Wherein:

[0116] The layout element acquisition module 310 is used to acquire complex document images; perform layout analysis on the complex document images to obtain layout analysis results; and perform data analysis processing on the layout analysis results to obtain a first set of layout elements.

[0117] The page information inference module 320 is used to divide the page range based on the first page element set using the minimum covering rectangle calculation method to obtain page structure information;

[0118] The column information inference module 330 is used to divide the column range based on the geometric expansion-intersection test and the page structure information using the connected subgraph solution method to obtain the column structure information.

[0119] The line break information inference module 340 is used to merge similar elements based on the column structure information using the element alignment method to obtain the line break structure information.

[0120] The inline information inference module 350 is used to split the covering elements based on the line structure information using the center point calculation method to obtain the inline structure information.

[0121] The multi-level information inference module 360 ​​is used to optimize information based on page structure information, column structure information, line structure information and inline structure information, and to identify the element relationships between lines and columns to construct multi-level structural information of complex document images.

[0122] Optionally, the page information inference module 320 is further used for:

[0123] Filter the non-main elements from the first set of elements to obtain the second set of elements.

[0124] The minimum covering rectangle is calculated based on the element set of the second page to obtain the element set of the first page.

[0125] Based on the first set of elements, the scope is divided according to the second set of elements to obtain the set of header elements, the set of footer elements, the set of left footnote elements, and the set of right footnote elements.

[0126] Based on the first set of heart element, header element, footer element, left footnote element, and right footnote element, obtain the page structure information.

[0127] Optionally, the column information inference module 330 is further used for:

[0128] The first set of core elements in the page structure information is combined with fine element filtering to obtain the second set of core elements.

[0129] Based on the second set of core elements, the geometric expansion-intersection test method is used to determine whether the core elements can be passed through, and the first set of core elements that can be passed through and the first set of core elements that cannot be passed through are obtained.

[0130] An undirected graph is constructed based on the first set of elements that can be displayed across the entire screen and the first set of elements that cannot be displayed across the entire screen, resulting in an undirected graph that can be displayed across the entire screen and an undirected graph that cannot be displayed across the entire screen. Based on the undirected graph that can be displayed across the entire screen and the undirected graph that cannot be displayed across the entire screen, the region is divided using the connected subgraph solution method, resulting in a second set of elements that can be displayed across the entire screen and a second set of elements that cannot be displayed across the entire screen.

[0131] Based on the horizontal overlap rule of elements, an undirected graph is constructed according to the second set of non-full-width elements to obtain a non-full-width filtering undirected graph; based on the non-full-width filtering undirected graph, the region is divided using the connected subgraph solution method to obtain a set of non-full-width single-column elements.

[0132] Based on the second set of elements that can be displayed across the entire column and the set of elements that cannot be displayed across the entire column, an undirected adjacency graph is constructed using a sorting and grouping method to obtain a column-based undirected adjacency graph; based on the column-based undirected adjacency graph, a secondary column division is performed using a connected subgraph solving method to obtain the second set of elements that can be displayed across the entire column and the second set of elements that cannot be displayed across the entire column.

[0133] Based on the second set of elements that can be displayed across the entire column and the second set of elements that cannot be displayed across the entire column, obtain the column structure information.

[0134] Optionally, the branch information inference module 340 is further used for:

[0135] Based on the column structure information, the column width alignment method is used to merge similar elements at the paragraph level to obtain a collection of various types of paragraph blocks;

[0136] Based on the collection of various types of paragraph blocks, the same type of elements at the row level are merged using the column height alignment method to obtain a collection of various types of row boxes;

[0137] The collection of various types of line boxes is determined as the line structure information.

[0138] Optionally, the inline information inference module 350 is further used for:

[0139] Based on the line structure information, obtain the set of text boxes and the set of formula boxes;

[0140] The center point is calculated based on the set of formula boxes to obtain the set of center points of the formula boxes;

[0141] Based on the set of center points of the formula box, the set of text boxes is split and validated to obtain the set of text boxes to be split and the set of text boxes not to be split.

[0142] Based on the set of text boxes to be split, and according to the set of formula boxes, select the formulas covered in the text boxes to obtain the set of covered formula boxes;

[0143] Based on the set of covered formula boxes, the text boxes to be split are split using the ascending horizontal coordinate sorting method to obtain the set of split text boxes;

[0144] Based on the split text box set, perform a merge check on the text boxes that are not split, and partially merge the text boxes of the same type to obtain the merged text box set and the unmerged text box set;

[0145] Obtain inline structure information based on the merged text box set and the unmerged text box set.

[0146] Optionally, the multi-level information inference module 360 ​​is further used for:

[0147] Based on the inline structure information, the minimum covering rectangle calculation method is used to optimize the structure information according to the line structure information to obtain the first optimized line structure information.

[0148] Based on the optimized row structure information, the minimum covering rectangle calculation method is used to optimize the structure information according to the column structure information to obtain the first optimized column structure information.

[0149] Based on the optimized column structure information, the minimum covering rectangle calculation method is used to optimize the structure information according to the page structure information to obtain the first optimized page structure information.

[0150] Based on a one-dimensional horizontal sorting rule, the in-row structural information is sorted by in-row elements to obtain the first optimized in-row structural information.

[0151] Based on the one-dimensional vertical sorting rule and the two-dimensional single-column sorting rule, the elements of the first optimized line structure information are sorted between rows to obtain the second optimized line structure information.

[0152] Based on the one-dimensional vertical sorting rule, the elements between columns of the first optimized column structure information are sorted to obtain the second optimized column structure information.

[0153] Based on the page layout order rules, the page elements of the first optimized page structure information are sorted to obtain the second optimized page structure information;

[0154] Based on the second optimized row structure information and the second optimized column structure information, the adjacent element type retrieval method is used to identify the merging relationship and obtain the element relationship information.

[0155] Based on the second optimized page structure information, the second optimized column structure information, the second optimized line structure information, the first optimized inline structure information, and the element relationship information, the multi-level structure information of the document image is obtained.

[0156] This invention proposes a method for inferring multi-level structural information from complex document images. Based on existing layout analysis, text detection, and formula detection models, it infers structural information at four levels—full-page, column, inline, and inline—from document images with complex layouts. It correctly determines the order, inclusion, and merging relationships between elements at each level, achieving accurate inference of multi-level structural information in complex documents. This provides a basis for downstream tasks such as text information extraction, significantly simplifying the complexity of downstream tasks. This invention can recursively parse structural information at four levels: full-page, column, block, and inline, and quickly infer the order and inclusion relationships of each element, improving the automated processing level of visually rich documents such as technical documents and academic papers. This invention is a real-time and efficient method for inferring multi-level result information from complex document images.

[0157] Figure 4 This is a schematic diagram of the structure of a multi-level information inference device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the multi-level structure information inference device may include the above-mentioned Figure 3 The illustrated device is a multi-level structural information inference apparatus for complex document images. Optionally, the multi-level structural information inference apparatus 410 may include a first processor 2001.

[0158] Optionally, the multi-level structure information inference device 410 may also include a memory 2002 and a transceiver 2003.

[0159] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0160] The following is combined with Figure 4 The various components of the multi-level structure information inference device 410 are described in detail below:

[0161] The first processor 2001 is the control center of the multi-level information inference device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0162] Optionally, the first processor 2001 can perform various functions of the multi-level information inference device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0163] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0164] In a specific implementation, as one example, the multi-level structure information inference device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0165] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0166] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and its interface circuitry for device 410 may be inferred through a multi-level structure information. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0167] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0168] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0169] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and the interface circuit of the device 410 can be inferred through multi-level structure information. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0170] It should be noted that, Figure 4 The structure of the multi-level information inference device 410 shown in the figure does not constitute a limitation on the router. The actual multi-level information inference device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0171] Furthermore, the technical effect of the multi-level structural information inference device 410 can be referred to the technical effect of the multi-level structural information inference method for complex document images described in the above method embodiments, and will not be repeated here.

[0172] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0173] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0174] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0175] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0176] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0177] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0178] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0180] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0183] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for inferring multi-level structural information of complex document images, characterized in that, The method includes: Acquire complex document images; perform layout analysis on the complex document images to obtain layout analysis results; perform data analysis and processing on the layout analysis results to obtain the first set of page elements; Based on the first set of elements, the page range is divided using the minimum covering rectangle calculation method to obtain page structure information; Based on the geometric expansion-intersection test, the column range is divided according to the page structure information using the connected subgraph solution method to obtain the column structure information; Based on the column structure information, use the element alignment method to merge similar elements to obtain the line structure information; Based on the row structure information, the center point calculation method is used to split the covering elements to obtain the in-row structure information; Information optimization is performed based on page structure information, column structure information, line structure information, and inline structure information. The element relationships between lines and columns are identified to construct multi-level structural information for complex document images.

2. The method for inferring multi-level structural information of complex document images according to claim 1, characterized in that, The step of dividing the page range based on the first set of page elements using the minimum covering rectangle calculation method to obtain page structure information includes: Filter the non-main elements from the first set of elements to obtain the second set of elements. The minimum covering rectangle is calculated based on the element set of the second page to obtain the element set of the first page. Based on the first set of elements, the scope is divided according to the second set of elements to obtain the set of header elements, the set of footer elements, the set of left footnote elements, and the set of right footnote elements. Based on the first set of heart element, header element, footer element, left footnote element, and right footnote element, obtain the page structure information.

3. The method for inferring multi-level structural information of complex document images according to claim 1, characterized in that, The geometric expansion-intersection test, based on page structure information, uses a connected subgraph solution method to divide the column range and obtain column structure information, including: The first set of core elements in the page structure information is combined with fine element filtering to obtain the second set of core elements. Based on the second set of core elements, the geometric expansion-intersection test method is used to determine whether the core elements can be passed through, and the first set of core elements that can be passed through and the first set of core elements that cannot be passed through are obtained. An undirected graph is constructed based on the first set of elements that can be displayed across the entire screen and the first set of elements that cannot be displayed across the entire screen, resulting in an undirected graph that can be displayed across the entire screen and an undirected graph that cannot be displayed across the entire screen. Based on the undirected graph that can be displayed across the entire screen and the undirected graph that cannot be displayed across the entire screen, the region is divided using the connected subgraph solution method, resulting in a second set of elements that can be displayed across the entire screen and a second set of elements that cannot be displayed across the entire screen. Based on the horizontal overlap rule of elements, an undirected graph is constructed according to the second set of non-full-width elements to obtain a non-full-width filtering undirected graph; based on the non-full-width filtering undirected graph, the region is divided using the connected subgraph solution method to obtain a set of non-full-width single-column elements. Based on the second set of elements that can be displayed across the entire column and the set of elements that cannot be displayed across the entire column, an undirected adjacency graph is constructed using a sorting and grouping method to obtain a column-based undirected adjacency graph; based on the column-based undirected adjacency graph, a secondary column division is performed using a connected subgraph solving method to obtain the second set of elements that can be displayed across the entire column and the second set of elements that cannot be displayed across the entire column. Based on the second set of elements that can be displayed across the entire column and the second set of elements that cannot be displayed across the entire column, obtain the column structure information.

4. The method for inferring multi-level structural information of complex document images according to claim 1, characterized in that, The step of merging similar elements using element alignment methods based on column structure information to obtain line structure information includes: Based on the column structure information, the column width alignment method is used to merge similar elements at the paragraph level to obtain a collection of various types of paragraph blocks; Based on the collection of various types of paragraph blocks, the same type of elements at the row level are merged using the column height alignment method to obtain a collection of various types of row boxes; The collection of various types of line boxes is determined as the line structure information.

5. The method for inferring multi-level structural information of complex document images according to claim 1, characterized in that, The step of splitting the covering elements using the center point calculation method based on the row structure information to obtain the in-row structure information includes: Based on the line structure information, obtain the set of text boxes and the set of formula boxes; The center point is calculated based on the set of formula boxes to obtain the set of center points of the formula boxes; Based on the set of center points of the formula box, the set of text boxes is split and validated to obtain the set of text boxes to be split and the set of text boxes not to be split. Based on the set of text boxes to be split, and according to the set of formula boxes, select the formulas covered in the text boxes to obtain the set of covered formula boxes; Based on the set of covered formula boxes, the text boxes to be split are split using the ascending horizontal coordinate sorting method to obtain the set of split text boxes; Based on the split text box set, perform a merge check on the text boxes that are not split, and partially merge the text boxes of the same type to obtain the merged text box set and the unmerged text box set; Obtain inline structure information based on the merged text box set and the unmerged text box set.

6. The method for inferring multi-level structural information of complex document images according to claim 1, characterized in that, The process of optimizing information based on page structure information, column structure information, line structure information, and in-line structure information, and identifying the element relationships between lines and columns to construct multi-level structural information for complex document images includes: Based on the inline structure information, the minimum covering rectangle calculation method is used to optimize the structure information according to the line structure information to obtain the first optimized line structure information. Based on the optimized row structure information, the minimum covering rectangle calculation method is used to optimize the structure information according to the column structure information to obtain the first optimized column structure information. Based on the optimized column structure information, the minimum covering rectangle calculation method is used to optimize the structure information according to the page structure information to obtain the first optimized page structure information. Based on a one-dimensional horizontal sorting rule, the in-row structural information is sorted by in-row elements to obtain the first optimized in-row structural information. Based on the one-dimensional vertical sorting rule and the two-dimensional single-column sorting rule, the elements of the first optimized line structure information are sorted between rows to obtain the second optimized line structure information. Based on the one-dimensional vertical sorting rule, the elements between columns of the first optimized column structure information are sorted to obtain the second optimized column structure information. Based on the page layout order rules, the page elements of the first optimized page structure information are sorted to obtain the second optimized page structure information; Based on the second optimized row structure information and the second optimized column structure information, the adjacent element type retrieval method is used to identify the merging relationship and obtain the element relationship information. Based on the second optimized page structure information, the second optimized column structure information, the second optimized line structure information, the first optimized inline structure information, and the element relationship information, the multi-level structure information of the document image is obtained.

7. A device for inferring multi-level structural information of complex document images, wherein the device is used to implement the method for inferring multi-level structural information of complex document images as described in any one of claims 1-6, characterized in that, The device includes: The layout element acquisition module is used to acquire complex document images; perform layout analysis on the complex document images to obtain layout analysis results; and perform data analysis and processing on the layout analysis results to obtain the first set of layout elements. The page information inference module is used to divide the page range based on the first page element set using the minimum covering rectangle calculation method, and obtain page structure information. The column information inference module is used to divide the column range based on the geometric expansion-intersection test and the page structure information using the connected subgraph solution method to obtain the column structure information. The line break information inference module is used to merge similar elements based on the column structure information using element alignment methods to obtain the line break structure information. The inline information inference module is used to split the covering elements based on the line structure information and the center point calculation method to obtain the inline structure information. The multi-level information inference module is used to optimize information based on page structure information, column structure information, line structure information and inline structure information, and to identify the element relationships between lines and columns to construct multi-level structural information of complex document images.

8. The multi-level structural information inference device for complex document images according to claim 7, characterized in that, The page information inference module is further used for: Filter the non-main elements from the first set of elements to obtain the second set of elements. The minimum covering rectangle is calculated based on the element set of the second page to obtain the element set of the first page. Based on the first set of elements, the scope is divided according to the second set of elements to obtain the set of header elements, the set of footer elements, the set of left footnote elements, and the set of right footnote elements. Based on the first set of heart element, header element, footer element, left footnote element, and right footnote element, obtain the page structure information.

9. A multi-level structural information inference device, characterized in that, The multi-level structure information inference device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images