Page analysis method and apparatus

Abstract

The present application relates to the technical field of computer, provide a kind of layout analysis method and device, the method comprises: extracting the image features of the document to be analyzed and the text features of layout analysis prompt text;Using image features and text features, generate the structured sequence containing the text content of each element in document and the logical order between elements;Extract the feature representation corresponding to each element in the structured sequence generation process;Based on the feature representation of each element, target detection is carried out on each element, and the coordinate position of each element is obtained.The present application decouples the layout analysis task into two stages of "logical order and content analysis" and "parallel element positioning", improves the inference efficiency of the whole layout analysis process, effectively solves the technical problem that efficiency and accuracy are difficult to consider in traditional method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a layout analysis method and device. BACKGROUND

[0002] Layout analysis refers to a technology of processing a document image, automatically recognizing and positioning different content elements such as text blocks, titles, tables, pictures and the like in the document, and analyzing the spatial layout and logical hierarchical relationship therebetween. Since it can convert an unstructured document image into structured data that can be processed by a computer, it is widely used in office automation, financial bill identification, intelligent document retrieval and knowledge base construction and the like.

[0003] At present, document layout analysis mainly adopts an end-to-end method based on a multi-modal large language model. The method usually inputs a document image into a multi-modal large model, and generates a structured sequence containing text content and coordinate positions of each layout element (such as a text block, a table and the like) in the document in an integrated manner through a self-recurrent manner. However, the self-recurrent mechanism needs to generate all content labels and coordinate position labels one by one, resulting in an excessively long output sequence and a serious inference time consumption, which is difficult to meet the efficient and real-time application requirements of layout analysis. SUMMARY

[0004] The present application provides a layout analysis method and device to solve the defects in the prior art.

[0005] The present application provides a layout analysis method, comprising the following steps:

[0006] extracting image features of a document to be analyzed and text features of layout analysis prompt text;

[0007] generating a structured sequence containing text content of each element in the document and logical order between the elements by using the image features and the text features;

[0008] extracting feature representations corresponding to each of the elements in a structured sequence generation process;

[0009] performing target detection on each of the elements based on the feature representations of each of the elements to obtain coordinate positions of each of the elements.

[0010] According to the layout analysis method provided by the present application, the structured sequence containing text content of each element in the document and logical order between the elements is generated by using the image features and the text features, comprising:

[0011] fusing the image features and the text features to generate a multi-modal input representation;

[0012] performing autoregressive decoding based on the multi-modal input representation, mapping the visual layout order of each of the elements provided by the image features to a logical order among the elements, and identifying and generating text content of each of the elements to obtain the structured sequence.

[0013] According to the page analysis method provided by the present application, the mapping of the visual layout order of each of the elements provided by the image features to a logical order among the elements, and the identification and generation of text content of each of the elements to obtain the structured sequence, comprises:

[0014] In the process of generating the text content of each of the elements in the logical order, a predefined structured marker is embedded at the corresponding position in the content stream, and the generated text content is wrapped to define the level and range of each of the elements, thereby obtaining the structured sequence.

[0015] According to the page analysis method provided by the present application, the extraction of the feature representation corresponding to each of the elements in the structured sequence generation process comprises:

[0016] Extracting the embedding vector corresponding to each of the word pieces in the structured sequence generation process;

[0017] Inputting the embedding vector corresponding to each of the word pieces into a mapping layer composed of a multi-layer perception machine and an activation function for processing to generate a mapped feature vector;

[0018] According to the element inclusion relationship defined by the start marker and the end marker in the structured sequence, all the mapped feature vectors belonging to the same element are combined to obtain the feature representation of the corresponding element.

[0019] According to the page analysis method provided by the present application, the target detection of each of the elements based on the feature representation of each of the elements to obtain the coordinate position of each of the elements comprises:

[0020] Combining all the feature representations of the same element into a query feature;

[0021] Performing target detection on each of the elements based on the query feature of each of the elements to obtain the coordinate position of each of the elements.

[0022] According to the page analysis method provided by the present application, the combining of all the feature representations of the same element into a query feature comprises:

[0023] For any element defined by the start marker and the end marker in the structured sequence, extracting the feature representation corresponding to each of the start marker, the end marker, and all the word pieces located between the start marker and the end marker;

[0024] merge the extracted overall feature representations to generate the query feature of any element.

[0025] According to the layout analysis method provided by the application, the image features of the document to be analyzed are extracted, which comprises:

[0026] segmenting the image of the document to be analyzed into a plurality of image blocks of a predetermined size;

[0027] flattening each image block into a vector, and combining the vectors corresponding to all image blocks with position encoding vectors to obtain an initial embedding representation, the position encoding vectors being used to represent the position of each image block in the image of the document to be analyzed;

[0028] encoding the initial embedding representation to obtain the image features of the document to be analyzed.

[0029] The application further provides a layout analysis device, comprising the following modules:

[0030] a first extraction unit configured to extract image features of a document to be analyzed and text features of a layout analysis prompt text;

[0031] an element analysis unit configured to generate a structured sequence containing the text content of each element in the document and the logical order between elements by using the image features and the text features;

[0032] a second extraction unit configured to extract feature representations corresponding to each element in the structured sequence generation process;

[0033] an element positioning unit configured to perform target detection on each element based on the feature representations of each element to obtain the coordinate position of each element.

[0034] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the layout analysis method as described above when executing the program.

[0035] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the layout analysis method as described above.

[0036] The application further provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the layout analysis method as described above.

[0037] The layout analysis method and device provided by the application decouples the layout analysis task into two stages of "logical sequence and content analysis" and "parallel element positioning", first generates a structured text sequence without coordinates, then extracts the feature representation in the generation process, and uses the feature representation to output the coordinate positions of all elements in parallel, realizing the unification of logical sequence recovery and efficient reasoning. Since the longest time-consuming coordinate generation task in the traditional method is separated from the serial autoregressive decoding process and innovatively transformed into a parallel processing target detection task, the logical sequence accurate understanding ability of complex document layout (such as multi-column, picture-text mixed layout, nested table) is completely retained, and the reasoning efficiency of the entire layout analysis process is greatly improved, effectively solving the technical problem that the traditional method cannot balance efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 It is a flowchart of the layout analysis method provided by the application.

[0040] Figure 2 It is a flowchart of another layout analysis method provided by the application.

[0041] Figure 3 It is a structural schematic diagram of the layout analysis device provided by the application.

[0042] Figure 4 It is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the application clearer, the technical scheme in the application will be described clearly and completely in combination with the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0044] Currently, document layout analysis mainly adopts large language models based on autoregression, methods based on target detection, and methods based on traditional rules or templates. Among them, the large language model method based on autoregression can better maintain the logical reading order of the document content through the sequential generation method, but the decoding characteristics of each character or mark lead to slow reasoning speed, which is difficult to meet the real-time analysis requirements. The method based on target detection can achieve efficient reasoning through parallel prediction, but the output of each layout element (such as text blocks and tables) is unordered, losing the original hierarchical structure and reading logic of the document. The method based on traditional rules or templates relies heavily on prior knowledge and is difficult to adapt to complex scenarios such as multi-column layout, irregular tables, and mixed text and graphics.

[0045] Therefore, the traditional text layout analysis method cannot simultaneously consider the accuracy of logical order and the efficiency of processing efficiency in the layout analysis task. To this end, the present application provides a layout analysis method, which can be applied to the automatic structural analysis and information extraction of various document images, such as processing scanned academic papers, financial statements, legal contracts, manuscripts, or various bills, etc. to identify and locate different layout elements such as titles, paragraphs, tables, pictures, etc. in them. Among them, Figure 1 is a flowchart of the layout analysis method provided by the present application, as Figure 1 shown, the method comprises steps 110, 120, 130 and 140.

[0046] Step 110, extract image features of the document to be analyzed and text features of the layout analysis prompt text.

[0047] Here, the document to be analyzed can be understood as a document that needs to be analyzed, which can be in the form of an image, or other formats of electronic documents, such as structured information containing text, vector graphics or embedded images, such as PDF, Word documents, etc. When processing, such electronic documents can be rendered or converted into image format for subsequent image feature extraction, which is not limited in this embodiment. Among them, the document to be analyzed can be a document scan obtained by a scanner, a high-speed camera, a document photo taken by a mobile phone, a camera, or obtained from a PDF, PPT, etc. For example, the document to be analyzed can be an academic paper image containing multi-column layout, mixed text and graphics, and nested tables.

[0048] Image features refer to vector representations extracted from the document to be analyzed, which can represent the visual content of the document to be analyzed (such as texture, shape, color, layout, etc. Information). Among them, the image features can be extracted by a backbone network such as a convolutional neural network (CNN) or a visual Transformer (Vision Transformer, ViT). For example, a pre-trained ResNet model can be used, the document to be analyzed is taken as input, the feature maps at different levels are obtained, and these feature maps are further processed (such as flattening, pooling, etc.) to generate the final image features. Considering the advantage of ViT model in capturing global dependence of image, the embodiment preferably uses ViT as visual encoder, that is, uses VIT to divide the image into a series of image blocks, and combines each image block after flattening with a position encoding vector, and inputs into the Transformer encoder, thereby obtaining image features that can finely depict global layout information.

[0049] The page analysis prompt text refers to a piece of natural language text used to guide the model to perform a specific page analysis task, which can be a Prompt text. Among them, the prompt text is used to clearly analyze the target and the expected output format. For example, for the page analysis of a paper, the prompt text can be "Please analyze the page structure of this document and identify the header, footer, title and body paragraphs"; for an invoice, the prompt text can be "Please extract all table items in the invoice."

[0050] Text features refer to vector representations obtained by processing the above page analysis prompt text using a text encoder, which can represent the semantic meaning of the prompt text. Among them, the text features can be extracted by a text encoder (Text Encoder), such as BERT (Bidirectional Encoder Representations from Transformers) or a dedicated prompt encoder (Prompt Encoder). For example, inputting the prompt text "Please analyze the page structure of this document" into the BERT model can obtain the corresponding text features.

[0051] Among them, the text features can be obtained based on the following formula:

[0052]

[0053] represents the text features obtained after processing by the encoder, represents the page analysis prompt text, represents the encoding of the page analysis text.

[0054] Step 120: Using image features and text features, generate a structured sequence containing the text content of each element in the document and the logical order between the elements.

[0055] Specifically, each element within a document can be understood as a component unit of the document being analyzed that has an independent function in terms of layout and content. For example, headers, footers, titles, paragraphs, tables, figures, and formulas can all be considered document elements.

[0056] The logical order between elements refers to the order that conforms to human reading habits or the inherent hierarchical relationship of a document, rather than a simple physical layout order from top to bottom or left to right. For example, for a two-column layout page, the logical order could be to read all the content in the left column first, and then read all the content in the right column; for a title and its paragraphs, the logical order is the title first, followed by the paragraphs.

[0057] A structured sequence is a special type of text sequence that not only contains the textual content of each element, but also explicitly represents the category, scope, and logical and hierarchical relationships between elements through predefined structured markers. For example, a structured sequence could be...<doc_s><header_s> Paper title< / header_e><page_s> This is the first paragraph.< / page_e><footer_s> Page 1_e>< / doc_e> .in,<doc_s> and<doc_e> These indicate the beginning and end of the document, respectively.<header_s> and<header_e> The header elements have been defined.

[0058] As an optional implementation, step 120 can be implemented using a multimodal large model. This model receives the image features and text features extracted in step 110 as input and effectively combines the two information through an internal fusion mechanism (such as cross-attention). Subsequently, the model generates tokens one by one in an autoregressive manner, ultimately forming a complete structured sequence. Because the model's internal mechanism can simultaneously understand the visual layout of the image and the instructions of the prompt text when generating the structured sequence, it can correctly map the visual layout relationship into a logical order that conforms to reading habits, and simultaneously complete the recognition of text within each element. The key to this step is that the generated structured sequence contains the text content of each element within the document and a structured sequence of the logical order between elements, but does not contain the coordinate information of each element. This greatly shortens the length of the output sequence, thereby significantly improving the inference speed.

[0059] The structured sequence can be obtained based on the following formula:

[0060]

[0061] wherein, denotes the structured sequence, denotes the textual feature, denotes the image feature, denotes the multi-modal large model.

[0062] Step 130, extracting the feature representation corresponding to each element in the structured sequence generation process.

[0063] Here, the feature representation can be understood as the hidden state or embedding vector corresponding to each token (whether it is a content token or a structured marker) in the structured sequence generated in the multi-modal large model of step 120. This feature representation is a highly condensed information carrier that integrates image visual information, text prompt information and previous content information relied on when generating the token.

[0064] As an optional embodiment, the hidden state vector of the top layer or a specific layer corresponding to the decoder when generating each token can be cached and recorded during the autoregressive generation process of step 120. For example, when the model generates the sequence <header_s>paper title< / header_e>, the internal feature vectors corresponding to the four tokens <header_s>, "paper", "title", and < / header_e> can be extracted respectively, and these extracted internal feature vectors are used as the feature representation of the "header" element.

[0065] Since these original feature vectors may have high dimensions or are not optimized for subsequent target detection tasks, this embodiment preferably processes these vectors through a mapping layer (Projection Layer) after extraction. For example, the mapping layer can be composed of one or more multi-layer perceptrons (MLP) and activation functions (such as ReLU or GeLU), which maps the original hidden state vector to a feature space more suitable as a target detection query (query), obtaining the mapped feature vector.

[0066] Step 140, performing target detection on each element based on the feature representation of each element, obtaining the coordinate position of each element.

[0067] Specifically, the coordinate position usually refers to the bounding box used to accurately locate the boundary of the element region on the original image, which can be represented by the coordinates of the top-left corner and the bottom-right corner (x1, y1, x2, y2) or by the coordinates of the top-left corner and the width and height (x, y, w, h).

[0068] The feature representation of each element is used as a "query feature" for driving the object detection. The feature representation of each element contains rich semantic information such as the category and content of the element, which can accurately guide the detection model to "find this specific element at which position in the image".

[0069] As an optional embodiment, a query-based object detection architecture such as DETR (DEtection TRansformer) or DINO model can be used to detect the elements. Specifically, for any element (such as a header) bounded by start and end tokens in the structured sequence, the feature representations corresponding to all tokens inside the element can be combined (such as summed, averaged, or concatenated) to form a single feature representation representing the complete element. Then, the feature representations of all elements are input into the decoder of the object detection model in parallel, and the model can output the coordinate positions of all elements simultaneously.

[0070] Since the feature representations of different elements are independent of each other, the object detection process can be completely parallelized, which is in sharp contrast to the traditional method of generating each coordinate value in a serial autoregressive manner. Considering the superior performance of the DINO model in detecting small and dense objects, in order to accurately locate the small text lines or closely arranged table cells that may exist in the document, the embodiment preferably uses DINO as the basic architecture of the element positioning module.

[0071] The layout analysis method provided by the embodiment decouples the layout analysis task into two stages of "logical order and content analysis" and "parallel element positioning", first generates a structured text sequence without coordinates, then extracts the feature representation in the generation process, and uses the feature representation to output the coordinate positions of all elements in parallel, realizing the unification of logical order recovery and efficient reasoning. Since the embodiment separates the time-consuming coordinate generation task in the traditional method from the serial autoregressive decoding process and innovatively transforms it into a parallelizable object detection task, it greatly improves the reasoning efficiency of the entire layout analysis process while retaining the ability to accurately understand the logical order of complex document layouts (such as multiple columns, mixed text and graphics, and nested tables), effectively solving the technical problem of balancing efficiency and accuracy in the traditional method.

[0072] Based on the above embodiment, the image features and text features are used to generate a structured sequence containing the text content of each element in the document and the logical order between the elements, including:

[0073] fusing the image features and the text features to generate a multi-modal input representation;

[0074] performing autoregressive decoding based on the multi-modal input representation to map the visual layout order of the elements provided by the image features to a logical order among the elements, and to identify and generate the text content of the elements, to obtain a structured sequence.

[0075] Specifically, the multi-modal input representation can be understood as a mixed feature vector sequence containing image visual information and text semantic instructions. Fusing the image features and the text features, the obtained multi-modal input representation contains both image features representing the visual content of the document to be analyzed and text features representing the instructions of the layout analysis task.

[0076] As an optional embodiment, the fusion process can be concatenating the image feature vector sequence obtained in step 110 and the text feature vector sequence. For example, if the dimension of the image features is [N, D] and the dimension of the text features is [M, D], the dimension of the multi-modal input representation obtained after concatenation is [N+M, D].

[0077] Further, in order to realize deeper interaction and alignment between the two modalities, the embodiment preferably adopts a more refined fusion strategy. For example, the image features and the text features can be jointly input into a multi-modal fusion module (such as one or more cross-attention layers of a Transformer). In this module, the text features can be used as queries (Query) to "focus" on the relevant areas in the image features; conversely, the image features can also be used as queries to "focus" on the key instructions in the text features. Finally, the output of this module is the multi-modal input representation after deep fusion and alignment.

[0078] Next, after obtaining the multi-modal input representation, autoregressive decoding is performed based on the multi-modal input representation to map the visual layout order of the elements provided by the image features to a logical order among the elements, and to identify and generate the text content of the elements, to obtain a structured sequence.

[0079] Here, autoregressive decoding is a sequential generation process, that is, when generating the output (a token) at the current time, it will depend on all previously generated outputs and the initial multi-modal input representation. This process is like "word by word" and "sentence by sentence" to build the final sequence, and the decision at each step is based on "the image that has been seen" and "the context that has been written".

[0080] In this embodiment, the decoder of the multi-modal large model can be used for autoregressive decoding, that is, after receiving the fused multi-modal input representation, the decoder of the multi-modal large model starts to perform autoregressive decoding, and its core tasks include logical order mapping and text content generation.

[0081] Where, for the logical order mapping, the model has learned rich layout knowledge through pre-training on massive document data. Therefore, when it “sees” the visual layout information implied in the multi-modal input representation (e.g., a document image with two columns), it will not mechanically generate text from top to bottom and left to right, but will be able to understand the inherent reading flow and intelligently map the visual layout order to the logical order that conforms to human reading habits. For example, the model will first generate the entire content of the left column, and then start generating the content of the right column, even if the top of the right column is physically higher than some parts of the left column.

[0082] For text content generation, as the model “moves” along the determined logical order, it will focus on the corresponding region of the image and perform an optical character recognition (OCR)-like function to recognize and convert the pixel information in the image into corresponding text tokens, one by one.

[0083] In this way, the model completes the understanding of the structure of the document, the reconstruction of the order, and the recognition of the content in one autoregressive decoding process, and finally outputs a structured sequence that contains accurate text content and reflects the correct logical order.

[0084] This embodiment can complete deep understanding of the visual layout of the document and accurate recognition of the text content in one step by deeply fusing image features and text features and using the autoregressive decoding capability of the multi-modal large model. This end-to-end approach not only simplifies the processing flow and avoids the need for separate layout analysis, region sorting, and OCR modules in traditional methods, but more importantly, it can handle complex scenarios that were previously difficult to solve (such as irregular text flow, cross-column tables, etc.), ensuring that the structured sequence generated has high accuracy in both logical order and text content.

[0085] Based on any of the above embodiments, mapping the layout order of each element provided by the image features to the logical order between elements, and recognizing and generating the text content of each element to obtain a structured sequence, includes:

[0086] In the process of generating the text content of each element in logical order, a predefined structured marker is embedded at the corresponding position in the content flow to wrap the generated text content to define the hierarchy and scope of each element, and a structured sequence is obtained.

[0087] Here, the predefined structured markers can be understood as a set of special tokens added to the vocabulary of the multimodal large model, which do not carry the semantic meaning of natural language by themselves, but serve as meta-instructions or labels to represent the structural information of the document. For example, <doc_s> and <doc_e> can be defined to represent the start and end of the document, respectively; <header_s> and <header_e> can represent the start and end of the header element, respectively; similarly, there can be <page_s> / <page_e> (main text page), <footer_s> / <footer_e> (footer), <table_s> / <table_e> (table), etc. /

[0088] By using pairs of start and end markers, the entire text content contained in any element can be accurately defined. For example, all the text tokens located between <header_s> and < / header_e> are explicitly attributed to the same header element.

[0089] By using nested markers, the inclusion relationship between elements can be clearly expressed. For example, the header element (wrapped by <header_s>... < / header_e>) and the footer element (wrapped by <footer_s>... _e>) can be nested in a page element (wrapped by <page_s>... < / page_e>) at the same time, accurately reflecting the tree-like hierarchical structure of the document.

[0090] As an optional embodiment, when the multimodal large model starts to generate a new logical element (e.g., the header of the document) according to its understanding of the layout of the image, it first generates the start marker of the element (e.g., <header_s>). Then, the model focuses on the corresponding header area in the image and generates the text content in this area one by one (e.g., "Chapter 1 Introduction"). When the text content of the element is completely generated, the model will immediately generate the corresponding end marker (e.g., < / header_e>). After that, the model will move to the next element (e.g., the main text paragraph) according to the logical order and repeat the process of "generating start marker -> generating content -> generating end marker".

[0091] For example, for a simple document page containing a header, main text and footer, the final structured sequence can be: out = <doc_s> <header_s>... < / header_e> <page_s>... < / page_e> <footer_s>... _e> < / doc_e>.

[0092] ​This embodiment intelligently embeds pairs of structured tags into the text content stream, transforming the document's two-dimensional physical layout and implicit logical hierarchy into a one-dimensional, self-contained, machine-readable text sequence. This not only makes the boundaries and affiliations of elements clear and unambiguous, but also provides the necessary foundation for the subsequent accurate extraction and aggregation of feature representations of each element.

[0093] Based on any of the above embodiments, the feature representations corresponding to each element in the structured sequence generation process are extracted, including:

[0094] Extract the embedding vectors corresponding to each word segmentation during the structured sequence generation process;

[0095] The embedding vectors corresponding to each word segment are input into a mapping layer consisting of a multilayer perceptron and an activation function for processing, generating a mapped feature vector.

[0096] Based on the element inclusion relationship defined by the start and end markers in the structured sequence, all mapped feature vectors belonging to the same element are merged to obtain the feature representation of the corresponding element.

[0097] Specifically, the embedding vector corresponding to each word segment can be understood as the internal hidden state vector generated by the multimodal large model during the autoregressive decoding process to generate each token (which can be understood as "word segment") in the structured sequence.

[0098] As an optional implementation, the hidden state vector corresponding to each token can be extracted as an embedding vector from the output of the last layer or several layers of the multimodal large model decoder. For example, for a sequence...<header_s> Paper title< / header_e> This step will extract the relevant data.<header_s> "Paper", "Title"< / header_e> These four tokens correspond one-to-one to four embedding vectors, denoted as [embed_h_s, embed_t1, embed_t2, embed_h_e].

[0099] After obtaining the embedding vectors corresponding to each word segment, the embedding vectors corresponding to each word segment are input into a mapping layer consisting of a multilayer perceptron and an activation function for processing to generate a mapped feature vector.

[0100] Here, the mapping layer can be understood as a lightweight neural network module whose main function is to transform and refine the feature space of the original embedding vectors. The original embedding vectors primarily serve the text generation task, and their feature distribution may not be entirely suitable as query input for subsequent object detection tasks. The mapping layer, through nonlinear transformations, can adjust these vectors into a more discriminative feature space that is more suitable for the localization task.

[0101] The structure of the mapping layer can be one or more multi-layer perceptron (MLP) composed of fully connected layers, and an activation function (such as ReLU, GeLU, Sigmoid, etc.) is used between layers to introduce nonlinearity. The feature mapping can be represented based on the following formula:

[0102]

[0103] wherein, represents the mapped feature vector, represents the embedding vector corresponding to the first token when the large language model (LLM) decodes to generate a structured sequence, represents the embedding vector corresponding to the first token when the large language model (LLM) decodes to generate a structured sequence, represents the embedding vector corresponding to the first token when the large language model (LLM) decodes to generate a structured sequence, represents a multi-layer perceptron, represents an activation function.

[0104] For example, the above four embedding vectors [embed_h_s, embed_t1, embed_t2, embed_h_e] are sent to the mapping layer one by one, and a new set of optimized feature vectors [Z_h_s, Z_t1, Z_t2, Z_h_e] is obtained.

[0105] Finally, according to the element inclusion relationship defined by the start and end markers in the structured sequence, all the mapped feature vectors belonging to the same element are merged to obtain the feature representation of the corresponding element.

[0106] Here, the element inclusion relationship is explicitly defined by pairs of structured markers (such as <header_s> and <header_e>). These markers clearly divide which consecutive tokens together constitute a complete element.

[0107] Considering that subsequent target detection requires a single, fixed-dimensional query feature to represent each element to be positioned, and an element itself is described by multiple, variable-number of token corresponding feature vectors, this embodiment merges all the mapped feature vectors belonging to the same element, so that the obtained feature representation can comprehensively summarize and represent the complete information of the element as a whole.

[0108] The merging operation can be element-wise addition of all mapped feature vectors belonging to the same element, or calculating the average of all mapped feature vectors belonging to the same element, or taking the maximum value in each dimension from all mapped feature vectors belonging to the same element, or connecting all vectors in sequence into a longer vector.

[0109] The embodiment converts the fine-grained token-level features generated during the decoding process into global element-level feature representations that can macroscopically represent each independent page element. Such element-level feature representations not only fuse the semantic information of all content within the element, but also implicitly contain the type (through the features of the marker) and range information of the element.

[0110] Based on any of the above embodiments, target detection is performed on each element based on the feature representation of each element to obtain the coordinate position of each element, including:

[0111] Combining all feature representations of the same element into a query feature;

[0112] Target detection is performed on each element based on the query feature of each element to obtain the coordinate position of each element.

[0113] Here, the query feature refers to a feature vector that can uniquely represent an element in the document to be analyzed, which guides the positioning process in target detection. Each query feature plays the role of a "search instruction", which is used to ask the model: "Please find the object (i.e., page element) in the image that matches this feature". A high-quality query feature should contain sufficient semantic information so that the model can accurately match it with a specific region in the image.

[0114] The mapped feature vectors (including the feature vectors corresponding to the content tokenization and the structured markers) belonging to the same element can be aggregated into a single vector through summation, averaging, etc. The vector is the query feature of the corresponding element. For example, the query feature of the header element is obtained by merging the mapped feature vectors of all tokens within it.

[0115] Next, target detection is performed on each of the elements based on the query feature of each element to obtain the coordinate position of each element. A target detection model can be used to implement this step. Considering the large size difference of elements in the document layout and the possible dense arrangement or even nesting, the embodiment preferably uses a query-based detection model, such as DINO (DETR with Improved Denoising Anchor Boxes).

[0116] Specifically, the query features (e.g., [Query_header, Query_page, Query_footer,...]) are input into the decoder of the DINO model in parallel, while the extracted image features are also input into the encoder and decoder of the DINO model.

[0117] Inside the decoder of DINO, each query feature interacts with the image features multiple times through cross-attention mechanisms. In this process, the query feature gradually "focuses" on the region of the image that best matches it.

[0118] After multiple layers of decoding, each query feature finally outputs two prediction results: one is the prediction of the element category (e.g., confirming that this is a "header"), and the other is the prediction of the coordinate position of the element in the image, usually represented as four coordinate values of a bounding box.

[0119] Since the decoder of the DINO model can handle all query features simultaneously, the coordinate positions of all elements in the document are calculated in parallel in one forward propagation, without generating coordinate values one by one like in autoregressive models, greatly improving the inference efficiency.

[0120] This embodiment can find the accurate positions of all elements simultaneously in one calculation by constructing corresponding query features for each element and using query features for target detection, avoiding the time bottleneck caused by serial decoding in traditional coordinate generation methods.

[0121] Based on any of the above embodiments, all feature representations of the same element are combined into query features, including:

[0122] For any element defined by the start and end markers in the structured sequence, the feature representations corresponding to the start marker, the end marker, and all word pieces between the start and end markers are extracted;

[0123] All extracted feature representations are merged to generate the query features of any element.

[0124] Specifically, the feature representations corresponding to the start and end markers (such as <header_s> and <header_e>) mainly carry the "category" information of the element. For example, the feature vector of <header_s> semantically points to "this is a header", which is crucial for distinguishing different types of elements.

[0125] The feature representation of all the words between the two (e.g., "paper title") mainly carries the content information of the element, which is crucial for distinguishing elements with different content but the same type (e.g., two different paragraphs).

[0126] Based on this, the feature representation of the start marker provides the clear class identity of the element, the feature representation of the end marker defines the end of the element content, and the feature representation of all the words between the two fills in the specific semantic details of the element, thereby constructing an overall portrait of the element with complete information dimension and specific class and content.

[0127] After obtaining all the feature representations, considering that the subsequent target detection needs a single, fixed-dimensional vector as a query input to locate a complete element, the embodiment merges all the extracted feature representations, thereby condensing multiple discrete feature information of an element into a unified overall feature representation, i.e., the obtained query feature is a high-dimensional aggregated vector that contains the class identity and specific content semantics of the element.

[0128] For example, when the entire document page needs to be located, the feature representations corresponding to <doc_s>, <doc_e>, and all other elements (e.g., header, body, footer, etc.) contained between them can be extracted for the outermost element defined by <doc_s> and <doc_e>, and then a query feature representing the entire page can be generated by merging (e.g., summing) the massive feature representations, which can be schematically represented as: query_doc = Z_<doc_s>+ Z_<header_s>+... + Z_<footer_e>+ Z_<doc_e>, where Z represents the mapped feature vector.

[0129] Based on any of the above embodiments, the image features of the document to be analyzed are extracted, including:

[0130] segmenting the image of the document to be analyzed into a plurality of image blocks of a predetermined size;

[0131] flattening each image block into a vector, and combining the vectors corresponding to all the image blocks with a position encoding vector to obtain an initial embedding representation, the position encoding vector being used to represent the position of each image block in the image of the document to be analyzed;

[0132] encoding the initial embedding representation to obtain the image features of the document to be analyzed.

[0133] Here, the image block of a predetermined size refers to a series of small image regions obtained by applying a fixed-size sliding window (without overlap) to the document corresponding image. For example, for an image of 224x224 pixels, if the predetermined size is set to 16x16 pixels, the image will be divided into (224 / 16)x(224 / 16)=196 image blocks.

[0134] Considering that after obtaining a plurality of image blocks, the subsequent flattening operation will lose the original two-dimensional spatial arrangement information of each image block, and the model itself for encoding the initial embedding representation does not have the ability to process the sequence order, it is necessary to add position information to the sequence composed of each image block. The position encoding vector is a vector with the same dimension as the flattened image block vector, which is learnable or fixed, and its value uniquely represents the absolute or relative position of each image block in the original image grid. Combining (usually element-wise addition) all flattened image block vectors with corresponding position encoding vectors can inject the spatial position information of each image block into its content representation, so that the initial embedding representation obtained can represent both the local visual content of each image block and its position information in the global image. Wherein, flattening to a vector means directly stretching each two-dimensional image block into a one-dimensional long vector.

[0135] Since the initial embedding representation contains both the visual content of each local region (i.e. image block) of the image and the spatial position information of each region in the global image through the position encoding, the image features obtained by encoding the initial embedding representation can fully capture the long-distance dependency relationship between any two regions in the image through the multi-head self-attention mechanism, form a comprehensive perception of the overall layout structure of the document (such as columns, margins, and position relationship between text and images), and then provide more information and more complete context visual input for subsequent multi-modal fusion and logical order analysis, significantly improving the analysis accuracy of complex layout documents.

[0136] Wherein, the initial embedding representation The initial embedding representation can be obtained based on the following formula:

[0137]

[0138]

[0139] Wherein, represents the th image block after segmentation, represents the th image block after segmentation, represents the th image block after segmentation, represents the total number of image blocks, represents an embedding matrix used to map a flattened image patch from pixel space to a high-dimensional feature space. represents a flattened image patch × image patch in pixel space, flattened into a ×C dimensional vector, and then projected into a D dimensional feature vector, represents a positional encoding vector.

[0140] Encoding the initial embedding representation can be represented by the following equation:

[0141]

[0142]

[0143]

[0144] wherein, represents the current level, represents the total number of Transformer encoder layers, represents the output of the th layer, which is the input of the th layer. represents a layer normalization operation, which is used to stabilize the training process and accelerate model convergence. MSA represents multi-head self-attention mechanism. represents the intermediate output obtained after the self-attention layer calculation of the th layer. MLP represents a multi-layer perceptron, usually composed of two linear layers and a nonlinear activation function (such as GELU). represents the final output of the th Transformer encoder layer. represents the 0th vector in the final output sequence . represents the final output sequence obtained after processing by all L encoder layers. represents the final image feature representing the entire document image to be analyzed, which is output by the entire visual encoder module.

[0145] Based on any of the above embodiments, Figure 2 is a flowchart of another page analysis method provided by the present application, as shown in Figure 2 , the method comprises:

[0146] First, the image of the document to be analyzed is input into a vision encoder, which segments the image into multiple image patches and converts them into a sequence of vision tokens, i.e., image features, that contain visual information of the image. Meanwhile, a layout analysis prompt text, which is used to guide the layout analysis task, is encoded into a sequence of text tokens, i.e., text features.

[0147] Next, the vision tokens and text tokens are input into a large language model (LLM), which starts generating a sequence of text in a self-recursive manner. At each token generation, the previous generation results and the fused multimodal features are referred to. The model embeds predefined structured markers, such as <doc_s> (document start), <header_s> (header start), and < / header_e> (header end), in the recognized text content stream during the generation process.

[0148] Finally, the output of this stage is a complete structured text sequence, such as out = <doc_s><header_s>...<header_e>...<doc_e>. The structured sequence contains not only the text content of all elements but also the logical relationships between elements through the nesting and order of markers.

[0149] In addition, the embedding vectors generated by the LLM during the generation of the structured sequence are extracted, with each embedding vector corresponding to each token of the output. These embedding vectors are input into a projection layer, which is usually an MLP, to transform them into a feature space more suitable for the target localization task, generating mapped feature vectors.

[0150] According to the inclusion relationship of the structured sequence, the mapped feature vectors are grouped. For example, all feature vectors corresponding from <header_s> to < / header_e> are grouped together, representing the "header" element. All feature vectors within each group are combined into a single, highly condensed feature vector through a fusion operation (such as summation or averaging), and this final vector is the feature representation of the element, i.e., the query feature.

[0151] The query features of all elements are input into a query-based target detection model (such as DINO) in parallel, and DINO takes these query features as "search instructions" to find the areas in the image features that best match each query. Since all queries are processed at the same time, DINO can output the coordinate positions corresponding to all elements at one time, such as doc_pos, header_pos, page_pos, etc. on the right side of the figure.

[0152] The page analysis device provided by the present application is described below. The page analysis device described below can be referred to in correspondence with the page analysis method described above.

[0153] Based on any of the above embodiments, Figure 3 The page analysis device provided by the present application is described below. The page analysis device described below can be referred to in correspondence with the page analysis method described above. Figure 3 As shown in the figure, the device comprises:

[0154] The first extraction unit 310 is configured to extract image features of the document to be analyzed and text features of the page analysis prompt text.

[0155] The element analysis unit 320 is configured to generate a structured sequence containing the text content of each element in the document and the logical order between elements by using the image features and the text features.

[0156] The second extraction unit 330 is configured to extract feature representations corresponding to each element in the structured sequence generation process.

[0157] The element positioning unit 340 is configured to perform target detection on each element based on the feature representation of each element to obtain the coordinate position of each element.

[0158] Based on any of the above embodiments, the structured sequence containing the text content of each element in the document and the logical order between elements is generated by using the image features and the text features, comprising:

[0159] Fusing the image features and the text features to generate a multi-modal input representation;

[0160] Based on the multi-modal input representation, performing autoregressive decoding to map the visual layout order of each element provided by the image features to the logical order between elements, and to recognize and generate the text content of each element to obtain the structured sequence.

[0161] Based on any of the above embodiments, the structured sequence is obtained by mapping the visual layout order of each element provided by the image features to the logical order between elements, and recognizing and generating the text content of each element, comprising:

[0162] In the process of sequentially generating the text content of each element in logical order, the predefined structured markers are embedded at the corresponding positions in the content stream to wrap the generated text content to define the hierarchy and scope of each element, obtaining a structured sequence.

[0163] Based on any of the above embodiments, the feature representation corresponding to each element in the structured sequence generation process is extracted, including:

[0164] The embedding vector corresponding to each word in the structured sequence generation process is extracted.

[0165] The embedding vector corresponding to each word is input into the mapping layer composed of a multi-layer perceptron and an activation function for processing to generate a mapped feature vector.

[0166] According to the element inclusion relationship defined by the start marker and the end marker in the structured sequence, all mapped feature vectors belonging to the same element are merged to obtain the feature representation of the corresponding element.

[0167] Based on any of the above embodiments, based on the feature representation of each element, target detection is performed on each element to obtain the coordinate position of each element, including:

[0168] All feature representations of the same element are combined into query features.

[0169] Based on the query features of each element, target detection is performed on each element to obtain the coordinate position of each element.

[0170] Based on any of the above embodiments, all feature representations of the same element are combined into query features, including:

[0171] For any element defined by the start marker and the end marker in the structured sequence, the feature representation corresponding to the start marker, the end marker, and all words located between the start marker and the end marker is extracted.

[0172] All extracted feature representations are merged to generate query features of any element.

[0173] Based on any of the above embodiments, the image features of the document to be analyzed are extracted, including:

[0174] The image of the document to be analyzed is segmented into multiple image blocks of a predetermined size.

[0175] Each image block is flattened into a vector, and the vectors corresponding to all image blocks are combined with a position encoding vector to obtain an initial embedding representation, the position encoding vector being used to represent the position of each image block in the image of the document to be analyzed.

[0176] The initial embedding representation is encoded to obtain the image features of the document to be analyzed.

[0177] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a layout analysis method, which includes: extracting image features of the document to be analyzed and text features of layout analysis prompt text; using the image features and text features, generating a structured sequence containing the text content of each element within the document and the logical order between the elements; extracting feature representations corresponding to each element during the generation of the structured sequence; and performing target detection on each element based on the feature representations of each element to obtain the coordinate positions of each element.

[0178] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the layout analysis method provided by the above methods. The method includes: extracting image features of the document to be analyzed and text features of layout analysis prompt text; using the image features and the text features, generating a structured sequence containing text content of each element in the document and the logical order between the elements; extracting feature representations corresponding to each element during the generation of the structured sequence; and performing target detection on each element based on the feature representations of each element to obtain the coordinate position of each element.

[0180] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a layout analysis method provided by any of the above methods, the method comprising: extracting image features of a document to be analyzed and text features of a layout analysis prompt text; generating a structured sequence containing text content of each element in the document and logical order between elements using the image features and the text features; extracting feature representations corresponding to each of the elements in the structured sequence generation process; and performing target detection on each of the elements based on the feature representations of each of the elements to obtain coordinate positions of each of the elements.

[0181] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0182] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0183] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A layout analysis method characterized by comprising: The method comprises: extracting image features of a document to be analyzed and text features of layout analysis prompt text; generating a structured sequence containing text content of each element in the document and logical order between elements by using the image features and the text features; extracting feature representations corresponding to each element in the structured sequence generation process; detecting each element based on the feature representations of each element to obtain the coordinate position of each element.

2. The layout analysis method according to claim 1, characterized by, The method of generating a structured sequence containing text content of each element in the document and logical order between elements by using the image features and the text features comprises: fusing the image features and the text features to generate a multi-modal input representation; based on the multi-modal input representation, performing autoregressive decoding to map the visual layout order of each element provided by the image features to the logical order between elements, and identify and generate the text content of each element, to obtain the structured sequence.

3. The layout analysis method according to claim 2, characterized by, The method of mapping the visual layout order of each element provided by the image features to the logical order between elements, and identifying and generating the text content of each element to obtain the structured sequence comprises: in the process of generating the text content of each element in turn according to the logical order, embedding a predefined structured marker at the corresponding position in the content stream to wrap the generated text content to define the level and range of each element, to obtain the structured sequence.

4. The page analysis method according to any one of claims 1 to 3, characterized in that, The method of extracting feature representations corresponding to each element in the structured sequence generation process comprises: extracting embedding vectors corresponding to each word in the structured sequence generation process; inputting the embedding vectors corresponding to each word into a mapping layer composed of a multi-layer perceptron and an activation function for processing to generate a mapped feature vector; according to the element inclusion relationship defined by the start marker and the end marker in the structured sequence, merging all the mapped feature vectors belonging to the same element to obtain the feature representation of the corresponding element.

5. The layout analysis method according to any one of claims 1 to 3, characterized by, The method of detecting each element based on the feature representations of each element to obtain the coordinate position of each element comprises: combining all the feature representations of the same element into a query feature; detecting each element based on the query feature of each element to obtain the coordinate position of each element.

6. The layout analysis method according to claim 5, wherein The method of combining all the feature representations of the same element into a query feature comprises: for any element defined by a start marker and an end marker in the structured sequence, extracting the feature representations corresponding to the start marker, the end marker, and all the words between the start marker and the end marker; merging all the extracted feature representations to generate the query feature of the any element.

7. The layout analysis method according to any one of claims 1 to 3, characterized by, The method of extracting image features of a document to be analyzed comprises: segmenting the image of the document to be analyzed into a plurality of image blocks of a predetermined size; flattening each image block into a vector, and combining the vectors corresponding to all the image blocks with a position encoding vector to obtain an initial embedding representation, the position encoding vector being used to represent the position of each image block in the image of the document to be analyzed; encoding the initial embedding representation to obtain image features of the document to be analyzed.

8. A layout analysis apparatus characterized by comprising: Comprising: a first extraction unit configured to extract image features of the document to be analyzed and text features of the layout analysis prompt text; an element analysis unit configured to generate a structured sequence containing text content of each element in the document and logical order between elements by using the image features and the text features; a second extraction unit configured to extract feature representations corresponding to each of the elements in the process of generating the structured sequence; an element positioning unit configured to perform target detection on each of the elements based on the feature representations of each of the elements to obtain coordinate positions of each of the elements.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to realize the layout analysis method of any one of claims 1-7. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the layout analysis method of any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the layout analysis method of any one of claims 1-7.

Images