A method for document content recognition and vertical section tree analysis

Abstract

The application discloses a method for document content recognition and vertical section tree structure analysis, which comprises the following steps: collecting a document and converting it into a picture, inputting a document element detection model to obtain document elements; the document element detection model comprises an input layer, an encoding layer and a decoding layer; five cross-stage local re-parameterization context modules are constructed in the input layer, and a dynamic multi-branch mixed convolution block is constructed in the encoding layer; a document hierarchical analysis model is constructed, the document elements are input into the document hierarchical analysis model after text content is extracted by an ocr tool, the final relationship probability between the document elements is obtained, the tree hierarchical structure and the sequence relationship between the document elements are obtained according to the final relationship probability, and thus a complete document is recognized. The application enhances the feature extraction capability of the document content by constructing the detection model and the analysis model, accurately captures the local structure relationship, and improves the detection precision of the document elements and the document content structure.

Description

Technical Field

[0001] This invention relates to the field of document recognition, and in particular to a method for document content recognition and analysis of the tree structure of document sections. Background Technology

[0002] In many fields, such as academic publishing, technical reports, and policy documents, a large number of documents carrying core knowledge exist in the form of digital documents with clear chapter structures. From a machine's perspective, these documents are often stored and circulated in a "visually oriented" format, and their internal semantic structure and chapter organization are not explicitly encoded in a machine-readable form. Specifically, the document content is presented as a series of visual elements arranged based on coordinates, and the hierarchical and sequential relationships between elements are implicit in the layout design, making it difficult for algorithms to automatically and accurately parse and utilize them.

[0003] Document content recognition and chapter structure analysis involves identifying various content elements from document images and inferring the hierarchical and sequential relationships between these elements to reconstruct the chapter logic tree. Document element detection and content extraction, as the primary and crucial step in structured processing, aims to locate, classify, and extract semantic visual blocks from document images. The output set of elements serves as the direct input for subsequent logical structure reconstruction, directly determining the upper limit of system performance.

[0004] Document element detection faces two major challenges: significant differences in the morphological scale of document elements and complex, heterogeneous document layouts. Both place high demands on detection models. Although general deep learning object detection models have been applied in this field, these models, originally designed for natural scenes, have insufficient receptive fields and feature representation capabilities, making it difficult to adapt to the diverse geometric shapes of document elements, thus exhibiting significant limitations.

[0005] In recent years, researchers have begun to focus on the parsing of complete hierarchical documents. DocParser was the first to build an end-to-end system for generating hierarchical document structures. However, its relation classification module heavily relies on hand-designed heuristics rather than trainable deep learning models, which limits the system's flexibility and generalization ability. Subsequent works such as HRDoc and Dochienet introduced deep learning-based models and emphasized the importance of handling multi-page documents. Nevertheless, existing methods still have room for improvement in effectively modeling long-range dependencies and complex interactions between elements across pages. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for document content recognition and main text section tree structure analysis.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A method for document content recognition and main text section tree structure analysis includes the following steps:

[0009] S1. Collect the document and convert it into an image format. Input the document image into the document element detection model to obtain the document elements.

[0010] The document element detection model comprises an input layer, an encoding layer, and a decoding layer. Five cross-stage locally reparameterizable context modules (CSP-RCB) are constructed in the input layer. The first and third CSP-RCB modules enhance the feature extraction capability of the document content. The features output by the second, fourth, and fifth CSP-RCB modules in the input layer are defined as shallow, mid-level, and deep features, respectively, and then input into the encoding layer to perceive the document's detailed information and high-level semantics. A dynamic multi-branch hybrid convolutional block (DIMB) is constructed in the encoding layer to achieve refined fusion and re-extraction of multi-scale features of the document. The output of the encoding layer is input into the decoding layer to obtain the document elements.

[0011] S2. Input the document elements into the document hierarchy parsing model to obtain the final relationship probability of the document elements. Based on the final relationship probability, obtain the tree hierarchy structure and order relationship between the document elements, thereby identifying the complete document.

[0012] The document hierarchy parsing model includes an embedding layer, an encoder, an interaction layer, and a gated fusion layer. The embedding layer adds page number embedding and element type embedding to obtain the input sequence. After the input sequence is encoded by the encoder, the interaction layer performs context enhancement on chapter elements and local window document elements. Based on the context enhancement of chapter elements and local window document elements, the gated fusion layer fuses the prediction results of local window document elements and chapter elements to obtain the final relationship probability.

[0013] In step S1, the CSP-RCB module includes a twelfth Conv layer, a first split layer, a fifth Concat layer, a fourteenth Conv layer, and two RCB layers, namely the first RCB layer and the second RCB layer.

[0014] The CSP-RCB module workflow is as follows:

[0015] Input feature map A into the twelfth Conv layer, adjust the number of channels of the feature map to match the input requirements of subsequent layers, and output feature map B;

[0016] Feature map B is input into the first split layer and split into two feature maps C and D;

[0017] Any feature map obtained after splitting the first split layer is input into the first RCB layer for deep feature extraction and enhancement of high-order semantic features, and a feature map E is output.

[0018] Input feature map E into the second RCB layer and output feature map F;

[0019] Feature maps C, D, E, and F are input together into the fifth Concat layer for concatenation. The original features are fused with the features extracted by the RCB layer in the channel dimension to achieve information aggregation, improve the overall representation ability, and output feature map G.

[0020] The feature map G is input into the fourteenth Conv layer to perform cross-channel information interaction of the feature maps and output the feature map H as the output of the CSP-RCB module.

[0021] The network layer architecture and parameters of the first RCB layer and the second RCB layer are the same, and the input and output feature map sizes are unchanged; the RCB layer includes the DW Conv layer, LayerNorm layer, Dilated RepBlock layer, SE layer and the thirteenth Conv layer;

[0022] The specific workflow of the RCB layer is as follows:

[0023] The feature map a is input into the DW Conv layer, and a depthwise separable convolution is performed to extract deep features from the feature map, outputting feature map b.

[0024] Add feature map a to feature map b to output feature map c;

[0025] Input feature map c into LayerNorm to stabilize and normalize the document element detection model during training, and output feature map d;

[0026] The feature map d is input into the Dilated RepBlock layer. While keeping the number of parameters and the output resolution unchanged, the receptive field is expanded and layout information over a larger range of the document page is integrated, so as to accurately understand the semantic role of each local region in the global document structure and output the feature map e.

[0027] The feature map e is input into the SE layer to model the interdependencies between channels, adaptively recalibrate the feature response weights of each channel, and output the feature map f.

[0028] The feature map f is input into the thirteenth Conv layer to perform cross-channel information interaction of the feature maps and output the feature map g.

[0029] Add feature map g to feature map c to output feature map h.

[0030] In step S1, the DIMB module includes a fifteenth Conv layer, a second split layer, a sixth Concat layer, an eighteenth Conv layer, and two DIDW modules, namely a first DIDW module and a second DIDW module.

[0031] The specific workflow of the DIMB module is as follows:

[0032] Feature map I is input into the fifteenth Conv layer to perform cross-channel information interaction of feature maps and output feature map J.

[0033] The feature map J is input into the second split layer for splitting, and two feature maps K and L are output.

[0034] Any feature map obtained from the second split layer is input into the first DIDW module, and semantic information extracted by dynamic multi-scale convolution is incorporated to adaptively fuse features from different sources and output feature map M.

[0035] Input the feature map M into the second DIDW module to extract multi-scale features and output the feature map N;

[0036] Feature maps K, L, M, and N are input together into the sixth Concat layer for concatenation along the channel dimension, which enhances the document element detection model's ability to represent diverse elements in document images and outputs feature map O.

[0037] The feature map O is input into the eighteenth Conv layer for dimensionality reduction, and the output feature map P is obtained.

[0038] The DIDW module includes a sixteenth Conv layer, a seventeenth Conv layer, a DIDWB module, a BatchNorm layer, and a Silu layer;

[0039] The specific workflow of the DIDW module is as follows:

[0040] Input feature map i into the sixteenth Conv layer for dimensionality reduction, and output feature map j;

[0041] Input feature map j into the DIDWB module to capture local and different features, perform weighted fusion of features, and output feature map k;

[0042] Input feature map i into the seventeenth Conv layer for dimensionality reduction, and output feature map l;

[0043] Add feature map k to feature map l to output feature map m;

[0044] The feature map m is sequentially fed into the BatchNorm layer and the Silu layer to stabilize the training of the document element detection model, and the feature map n is output.

[0045] The specific process of the DIDWB module is as follows:

[0046] The feature map [B,C,H,W] is input into three depthwise separable convolutional layers, each outputting a feature map of [B,C,H,W]; where B is the batch size, C is the number of channels, H is the height, and W is the width; the three depthwise separable convolutional layers are horizontal bar convolution, square convolution, and vertical bar convolution; the square convolution is responsible for capturing local, square feature patterns; the horizontal bar convolution captures long-range dependencies in the horizontal direction; and the vertical bar convolution captures contextual information in the vertical direction.

[0047] The feature map [B,C,H,W] is input into the AvgPool layer for average pooling, outputting a feature map [B,C,1,1]. This output feature map is then fed into a 1x1 convolution to obtain a 3C-dimensional vector [B,3C,1,1]. The shape is then reshaped to [3,B,C,1,1]. Softmax function is applied to the first dimension for normalization, resulting in three sets of weights. Each set of weights corresponds to the output of a convolutional branch. Corresponding to the horizontal bar convolution branch, Corresponding to the square convolution branch, Corresponding vertical bar convolution branch;

[0048] After weighting and fusing the outputs of the three sets of weights with the outputs of the three corresponding depthwise separable convolutional layers, the resulting feature maps are activated by BatchNorm and Silu respectively, and the shape of the output remains unchanged, still being a feature map of [B,C,H,W].

[0049] In step S1, the document elements are specifically as follows:

[0050] Given a document D consisting of P pages, represented as a sequence of page images:

[0051] ;

[0052] in, This represents the image on the Pth page;

[0053] Based on document D, a document element detection model Detect and categorize document elements on the page:

[0054] ;

[0055] in, , This indicates the page number on page p. The x and y coordinates of the top-left corner of the element's bounding box. This indicates the page number on page p. The x and y coordinates of the bottom right corner of the element's bounding box. Category tags for document elements, including Title, Text, Figure, and Table; This represents the total number of document elements detected on page p.

[0056] Through optical character recognition engine Extract text content from text-type elements in a document:

[0057] ;

[0058] For non-text elements, their text content Set as tag type ;

[0059] The final output is a flattened collection containing all elements from all pages. :

[0060] ;

[0061] Each document element is represented by a tuple. This represents all of its attributes:

[0062] ;

[0063] Where p represents the document page number.

[0064] In step S2, the embedding layer is specifically as follows:

[0065] For page p of the document, divide according to coordinates Sort the document elements by ascending order of their y-coordinates (vertical coordinates), and by ascending order of their x-coordinates (horizontal coordinates) if they are the same y-coordinate, to obtain an ordered list. ;

[0066] For document elements Using a pre-trained word segmenter Segmented into a sequence of subwords, non-text elements use their category information. As text fill Get the token list ,in This is the length of the tokens in this element; all tokens within the same element share the element's bounding box. ; Arrange the obtained list of tokens belonging to each document element within the page into an ordered list Arrange them in order, then add [CLS] and [SEP] to get That is, the input sequence of the layoutlmv3 model. ,in, This represents the total number of tokens on page p.

[0067] Here, the embedding of a token in a document element is represented as follows: , It integrates text, location, visual, page, and type information, and the specific calculation is as follows:

[0068] ;

[0069] in, This represents the input embedding representation of the k-th token in the p-th document element of the page. For text embedding, the element text is converted into a token ID sequence using a tokenizer, and then the token is retrieved using the layoutlmv3 token embedding matrix. For 1D positional embedding, it is obtained by positional encoding based on the position index of each token in the input sequence; For 2D position embedding, the layout embedding layer is obtained by assigning a corresponding bounding box to each token. For visual embedding, the page image is divided into fixed-size patches, which are then flattened and obtained through linear projection. Embed the page number for page P; Embedded for element type;

[0070] The page number embedding is calculated as follows:

[0071] Given a page number to which a token belongs Using the sine-cosine position coding function Will After being transformed into a continuous vector, it is then passed through a linear projection layer. The sinusoidal encoding is mapped to a space aligned with the hidden layer dimensions of the document hierarchy parsing model, as follows:

[0072] ;

[0073] The element type embedding is calculated by assigning a predefined category label to each document element. Maintain document element type embedded lookup table ,in, The total number of types, For the embedding dimension; for type is The element type embedding for each token within a document element is obtained through a direct table lookup. .

[0074] In step S2, the encoder encodes the document page by page, inputting a sequence. After inputting the layoutlmv3 encoder, we obtain the context-aware hidden state of each token:

[0075] ;

[0076] Feature aggregation of document elements:

[0077] Split the p-th page of the document into The document element, let the first Page 1 The set of token indexes contained in each document element is The token representation within the document element is aggregated using average pooling to obtain a compact vector of the document element. The specific calculations are as follows:

[0078] ;

[0079] Iterate through all the pages in the document to get There are 1 document element, P is the total number of pages; based on the compact vector of document elements. By arranging the document elements into a sequence based on the page numbers and vertical coordinates within the page in the original document, a global element representation sequence is constructed. The specific calculations are as follows:

[0080] ;

[0081] in, For the i-th globally sorted document element, For embedded dimensions; the sorting rules are as follows: first, sort by page number p in ascending order; for elements within the same page, sort by the coordinates of the document elements within the page, that is, first sort by the vertical coordinate y in ascending order, and if y is the same, then sort by the horizontal coordinate x in ascending order.

[0082] In step S2, the interaction layer is:

[0083] Given a global element representing a sequence Candidate chapter title elements are filtered based on predefined element type tags. ; ,in, Number of candidate chapter titles For the embedded dimension;

[0084] Calculate the semantic relationships between candidate chapter title elements, mapping them to the query space Q, key space K, and value space V. The specific calculation is as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] in, These are learnable parameters;

[0089] Based on the query space Q and the key space K, the mutual attention among candidate chapter titles is calculated. The specific calculation formula is as follows:

[0090] ;

[0091] Where T is the transpose. For projection dimensions;

[0092] Based on the mutual attention level A, the attention output is:

[0093] ;

[0094] Projection through attention output By performing residual connections and layer normalization, we obtain a context-enhanced chapter element representation. Specifically:

[0095] ;

[0096] in, This is the normalization function;

[0097] Sum the values ​​in each column of mutual attention A to obtain the importance score for each candidate chapter title. :

[0098] ;

[0099] The total important score for all candidate chapter titles is: ;

[0100] Normalize the total importance scores of all candidate chapter titles to obtain the importance distribution. :

[0101] ;

[0102] Based on α, the Top-K candidate nodes with the highest scores are selected as chapter boundary nodes; the chapter boundary nodes divide the document into K+1 local semantic windows; each local semantic window defined by the chapter boundary node... A primary semantic unit of the corresponding document;

[0103] For local semantic windows Document elements within are represented as The enhanced representation is obtained through an L-layer Transformer encoder. The specific calculations are as follows:

[0104] ;

[0105] in, d represents the local semantic window, and d represents the embedding dimension;

[0106] After all local semantic window interactions are completed, the enhanced document element representation is obtained as follows: :

[0107] ;

[0108] Where N is the total number of document elements and d is the embedding dimension.

[0109] In step S2, the gated fusion layer is:

[0110] Based on enhanced document elements Construct joint representation vector Where i, j represent the chapter element pair (i, j); Inputting the data into a multilayer perceptron classifier (MLP) yields a global relation prediction head, i.e., a relation matrix. The specific calculation formula is as follows:

[0111] ;

[0112] MLP stands for Multilayer Perceptron; The dimension is r, where r is the relation type;

[0113] A dual-branch gating fusion mechanism is designed, consisting of local branches and chapter branches. The local branches utilize global relationships to predict the prediction results for head-to-chapter element pairs. The specific calculation is as follows:

[0114] ;

[0115] in, That is, the relation matrix 'r' represents the relation type. For the index set of candidate chapter title elements , Number of candidate chapter titles;

[0116] Chapter branches are represented by context-enhanced chapter elements. Predict the relation by constructing a joint representation vector for each pair of chapter elements (i,j). Input it into the multilayer perceptron The prediction results of chapter branches are obtained. ;

[0117] Gated networks dynamically generate fusion weight vectors for each relation type based on the context of local and chapter branches. :

[0118] ;

[0119] in, and For learnable parameters, For activation functions; They are respectively Features of the i-th and j-th chapter elements; They are respectively Features of the i-th and j-th chapter elements;

[0120] The final inter-chapter relationship prediction is obtained through weighted fusion:

[0121] ;

[0122] Will Write back the relation matrix of all elements Calculate the probability of the final relationship:

[0123] ;

[0124] Employing multi-label binary cross-entropy loss The training process is performed, and a mask is applied to avoid invalid pairings, as follows:

[0125] ;

[0126] in, For real labels, i and j represent chapter element pairs (i, j), r is the relation type, and R is the total number of relation types; Here, is the Sigmoid function, and N is the total number of document elements.

[0127] Meanwhile, this invention provides:

[0128] A server includes a processor and a memory, the memory storing at least one program that is loaded and executed by the processor to implement the above-described method for document content recognition and main text section tree structure analysis.

[0129] A computer-readable storage medium storing at least one program, which is loaded and executed by a processor to implement the above-described method for document content recognition and main text section tree structure analysis.

[0130] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0131] 1. The document element detection model of this invention constructs a CSP-RCB module and a DIMB module. The CSP-RCB module enhances feature extraction capabilities and expands the receptive field, while the DIMB module has the ability to adapt to element shapes. The combination of the two enables the model to robustly improve the detection accuracy of various elements such as text, titles, charts, and formulas.

[0132] 2. The document hierarchy parsing model of this invention adopts the strategy of "page-by-page encoding and element aggregation", which avoids the sequence length limitation while preserving the complete visual-linguistic context within the page.

[0133] 3. The document hierarchy parsing model of this invention introduces page number embedding and element type embedding in the embedding layer, injecting the model with key prior knowledge of page order awareness and element semantic function differentiation.

[0134] 4. The document hierarchy parsing model interaction layer of this invention adaptively filters chapter boundaries, divides long documents into semantically cohesive local windows, and performs efficient attention interaction within the windows, achieving a significant reduction in computational complexity and an effective improvement in relation modeling accuracy.

[0135] 5. This invention provides improved performance in predicting document chapter relationships by fusing local branch and chapter branch gating. Attached Figure Description

[0136] Figure 1 This is a diagram of the overall architecture of the document element detection model.

[0137] Figure 2 The Stem layer architecture diagram for the document element detection model.

[0138] Figure 3 The architecture diagram of the CSP-RCB module for document element detection model.

[0139] Figure 4 This is a diagram of the DIMB module architecture for the document element detection model.

[0140] Figure 5 The architecture diagram of the DIDWB module for document element detection model.

[0141] Figure 6 This is a flowchart of the overall document hierarchy parsing model.

[0142] Figure 7 This is a schematic diagram of the interactive layer of the document hierarchy parsing model.

[0143] Figure 8This is a schematic diagram of the gated fusion layer of the document hierarchy parsing model. Detailed Implementation

[0144] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0145] A method for document content recognition and main text section tree structure analysis includes the following steps:

[0146] S1. Collect the document and convert it into an image format. Input the document image into the document element detection model to obtain the document elements.

[0147] The document element detection model comprises an input layer, an encoding layer, and a decoding layer. Five cross-stage locally reparameterizable context modules (CSP-RCB) are constructed in the input layer. The first and third CSP-RCB modules enhance the feature extraction capability of the document content. The features output by the second, fourth, and fifth CSP-RCB modules in the input layer are defined as shallow, mid-level, and deep features, respectively, and then input into the encoding layer to perceive the document's detailed information and high-level semantics. A dynamic multi-branch hybrid convolutional block (DIMB) is constructed in the encoding layer to achieve refined fusion and re-extraction of multi-scale features of the document. The output of the encoding layer is input into the decoding layer to obtain the document elements.

[0148] S2. Input the document elements into the document hierarchy parsing model to obtain the final relationship probability of the document elements. Based on the final relationship probability, obtain the tree hierarchy structure and order relationship between the document elements, thereby identifying the complete document.

[0149] The document hierarchy parsing model includes an embedding layer, an encoder, an interaction layer, and a gated fusion layer. The embedding layer adds page number embedding and element type embedding to obtain the input sequence. After the input sequence is encoded by the encoder, the interaction layer performs context enhancement on chapter elements and local window document elements. Based on the context enhancement of chapter elements and local window document elements, the gated fusion layer fuses the prediction results of local window document elements and chapter elements to obtain the final relationship probability.

[0150] Step S1 is that document elements have strong contextual dependencies. To determine the semantic role of a document element (such as "figure title"), it is necessary to rely on its contextual relationship with surrounding elements (such as illustrations). Moreover, document elements vary greatly in scale and form: there are long and thin title lines, large charts and graphs, as well as blocky paragraphs. Different elements have different requirements for receptive field and feature form.

[0151] To address the characteristics of document images, such as strong element layout dependencies and significant differences in scale and geometric shape, the core components of the DEIM object detection framework are enhanced, such as... Figure 1We propose a document element detection model based on enhanced context awareness and adaptive multi-branch convolution.

[0152] The existing HGNetV2 backbone and RepNCSPELAN fusion module, due to their fixed-size convolutional kernels and limited receptive field, struggle to simultaneously meet the aforementioned requirements. Therefore, this invention proposes two core innovative modules: the CSP-RCB module and the DIMB module. Through CSP-RCB, an enhanced backbone network with a wider receptive field and stronger feature extraction capabilities is constructed. Through DIMB, this invention replaces the fixed-shape convolutions in the original fusion module, enabling the feature fusion network to adapt to the size and shape of document elements.

[0153] In stage 1-2, a CSP-RCB is used to quickly extract shallow features. A single module ensures efficiency by using the CSP structure to pass some features directly through shortcut connections, while the other part is used for deep feature extraction.

[0154] In Stage 3, two CSP-RCB modules are used, and after downsampling, the feature map size is halved. Stacking two CSP-RCB modules here leverages their core advantage—dilated RepBlock layers—to perform large-scale context modeling. Concatenating the two modules is equivalent to significantly expanding the receptive field twice, effectively integrating layout information across a wider range of the page. This is crucial for understanding document elements; for example, determining whether a text block is a "heading" or "body text" requires considering the surrounding paragraphs and page structure. The stacking of the two modules enables the network to capture a large range of document context, meeting the high demands of contextual understanding in document element detection. Simultaneously, the channel attention mechanism (SE layer) recalibrates the features at the end of each module, selecting the feature channels most important for the current stage's classification task.

[0155] In Stage 4, a CSP-RCB is used for feature refinement and computational efficiency optimization. After deeper downsampling, the feature map resolution is further reduced, resulting in high-level semantic features. At this point, the features already contain rich context. Therefore, reducing the number of modules to one is primarily aimed at the final refinement of the existing deep semantic features.

[0156] The concatenation of the outputs of the second, fourth, and fifth CSP-RCBs forms the basis for constructing the feature pyramid. These three outputs correspond to feature maps of different depths and resolutions in the backbone network. The second CSP-RCB (shallow layer) contains rich detailed information; the fourth CSP-RCB (mid-layer) includes deeper features and some contextual information; and the fifth CSP-RCB (deep layer) contains high-level semantics and rich context. Concatenating these features and inputting them into subsequent encoding layers provides the "raw material" for multi-scale fusion, enabling subsequent modules to simultaneously perceive detailed information and high-level semantics.

[0157] The second concat concatenation joins the features processed by the first DIMB (containing high-level semantics) and the features processed by convolution in p3 (containing detailed information). The resulting feature map contains both high-level semantic information and detailed information. This concatenated feature map is then input into the second DIMB for further feature extraction (ultimately, the feature map, equivalent to p3 size, incorporates high-level semantic information from deeper layers). The eighth and fifth concatenation operations learn the correlations between channels in the feature maps processed by DIMB and AIFI, recombine the input features into a new feature space.

[0158] The document image of [B,3,H,W] is input into the Stem layer for downsampling and preliminary feature extraction, and the feature map of [B,16,H / 4,W / 4] is output.

[0159] Input the feature map [B, 16, H / 4, W / 4] into the first CSP-RCB module and output the feature map [B, 64, H / 4, W / 4].

[0160] The feature map [B, 64, H / 4, W / 4] is input into the first Conv layer [Conv2d(64, 64, kernel_size=3, stride=2, padding=1, groups=64 (representing depthwise separable convolution), bias=False), BatchNorm2d(64), no activation (use_act=False)] for downsampling, and the feature map [B, 64, H / 8, W / 8] is output;

[0161] Input the feature map [B,64,H / 8,W / 8] into the second CSP-RCB module and output the feature map [B,256,H / 8,W / 8], which is the input of P3;

[0162] The feature map [B,256,H / 8,W / 8] is input into the second Conv layer [Conv2d(256,256,kernel_size=3,stride=2,padding=1,groups=256),BN(256)] for downsampling, and the feature map [B,256,H / 16,W / 16] is output.

[0163] Input the feature map [B,256,H / 16,W / 16] into the third CSP-RCB module, and output the feature map [B,512,H / 16,W / 16].

[0164] Input the feature map [B,512,H / 16,W / 16] into the fourth CSP-RCB module, and output the feature map [B,512,H / 16,W / 16], which is the feature map input to P4.

[0165] The feature map [B,512,H / 16,W / 16] is input into the third Conv layer [Conv2d(512,512,kernel_size=3,stride=2,padding=1,groups=512),BN(512)] for downsampling, and the feature map [B,512,H / 32,W / 32] is output.

[0166] Input the feature map [B,512,H / 32,W / 32] into the fifth CSP-RCB module, and output the feature map [B,1024,H / 32,W / 32], which is the feature map input by P5;

[0167] The feature map [B,256,H / 8,W / 8] output by the second CSP-RCB module is input into the seventh Conv layer [Conv2d(256,256,kernel_size=1,stride=1,padding=0,bias=False),BN(256)] for projection, and the feature map [B,256,H / 8,W / 8] is output.

[0168] The feature map [B, 512, H / 16, W / 16] output by the fourth CSP-RCB module is input into the sixth Conv layer [Conv2d(512, 256, kernel_size=1, stride=1, padding=0, bias=False), BN(256)] for projection, and the feature map [B, 256, H / 16, W / 16] is output.

[0169] The feature map [B, 1024, H / 32, W / 32] output by the fifth CSP-RCB module is input into the fourth Conv layer [Conv2d(1024, 256, kernel_size=1, stride=1, padding=0, bias=False), BN(256)] for projection, and the feature map [B, 256, H / 32, W / 32] is output.

[0170] The feature map [B,256,H / 32,W / 32] output from the fourth Conv layer is input into the AIFI layer, i.e., the Transformer coding layer, and the output feature map [B,256,H / 32,W / 32] is encoded, with the input and output remaining unchanged;

[0171] Input the feature map [B,256,H / 32,W / 32] output by the AIFI layer into the fifth Conv layer [Conv2d(256, 256, kernel_size=1, stride=1, padding=0), BN(256)], and output the feature map [B,256,H / 32,W / 32].

[0172] The feature map [B,256,H / 32,W / 32] output from the fifth Conv layer is input into the first Upsample layer for upsampling, and the feature map [B,256,H / 16,W / 16] is output.

[0173] The feature maps [B,256,H / 16,W / 16] output from the first Upsample layer and the sixth Conv layer are input into the first Concat layer for concatenation along the channel dimension, outputting a feature map [B,512,H / 16,W / 16].

[0174] The feature map [B,512,H / 16,W / 16] output from the first Concat layer is input into the first DIMB module for multi-scale feature fusion, and the feature map [B,256,H / 16,W / 16] is output.

[0175] The feature map [B,256,H / 16,W / 16] output by the first DIMB module is input into the eighth Conv layer [Conv2d(256,256,kernel_size=1,stride=1),BN(256)], and the feature map [B,256,H / 16,W / 16] is output.

[0176] The feature map of [B,256,H / 16,W / 16] is input into the second Upsample layer for upsampling, and the feature map of [B,256,H / 8,W / 8] is output.

[0177] The feature maps [B,256,H / 8,W / 8] output from the second Upsample layer and the seventh Conv layer are input into the second Concat layer for channel concatenation, and the feature map [B,512,H / 8,W / 8] is output.

[0178] Input the feature map [B,512,H / 8,W / 8] into the second DIMB module, and output the feature map [B,256,H / 8,W / 8].

[0179] The feature map [B,256,H / 8,W / 8] output by the second DIMB module is input into the tenth Conv layer [Conv2d(256,256,1,1)+Conv2d(256,256,kernel_size=3,stride=2,groups=256),BN(256)] for downsampling convolution, and the feature map [B,256,H / 16,W / 16] is output.

[0180] The feature maps [B,256,H / 16,W / 16] output from the 10th Conv layer and the 8th Conv layer are input into the 3rd Concat layer for concatenation along the channel dimension, outputting a feature map [B,512,H / 16,W / 16].

[0181] Input the feature map [B,512,H / 16,W / 16] into the third DIMB module, and output the feature map [B,256,H / 16,W / 16].

[0182] The feature map [B,256,H / 16,W / 16] is input into the eleventh Conv layer [Conv2d(256,256,1,1)+Conv2d(256,256,kernel_size=3,stride=2,groups=256),BN(256)] for downsampling convolution, and the feature map [B,256,H / 32,W / 32] is output.

[0183] The feature maps [B,256,H / 32,W / 32] output from the fifth and eleventh Conv layers are input into the fourth Concat layer for concatenation along the channel dimension, outputting a feature map [B,512,H / 32,W / 32].

[0184] Input the feature map [B,512,H / 32,W / 32] into the fourth DIMB module, and output the feature map [B,256,H / 32,W / 32].

[0185] The [B,256,H / 8,W / 8] feature map output by the second DIMB module, the [B,256,H / 16,W / 16] feature map output by the third DIMB module, and the [B,256,H / 32,W / 32] feature map output by the fourth DIMB module are concatenated along the length dimension and then input into the decoder for decoding. The final output is the category label and bounding box of the document element in the document image.

[0186] Among them, such as Figure 2 The Stem layer consists of five ConvBNAct layers, zero-padding, max pooling layers, and Concat layers. Each ConvBNAct layer includes a two-dimensional convolutional layer Conv2d, a two-dimensional batch normalization layer BatchNorm2d, and a corrected linear unit ReLU.

[0187] The document image of [B,3,H,W] is input into the first ConvBNAct layer [Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=2, padding=1, groups=1, bias=False), BatchNorm2d(16), ReLU] for downsampling and preliminary feature extraction, and outputs a feature map of [B,16,H / 2,W / 2], where B is the batch number, 3 is the number of channels, H is the height of the document image, W is the width of the document image; in_channels are the input channels, out_channels are the output channels, kernel_size is the kernel size, stride is the stride, padding is the padding, groups=1 represents ordinary convolution, not depthwise separable convolution, and bias is the bias.

[0188] Zero-padding the feature map [B,16,H / 2,W / 2] with F.pad(0,1,0,1) results in the feature map [B,16,H / 2+1,W / 2+1].

[0189] The feature map [B, 16, H / 2+1, W / 2+1] is input into the second ConvBNAct layer [Conv2d(16, 8, kernel_size=2, stride=1, padding=0), BN(8), ReLU], and the output feature map [B, 8, H / 2, W / 2] is then zero-padding to obtain the feature map [B, 8, H / 2+1, W / 2+1].

[0190] The feature map [B,8,H / 2+1,W / 2+1] is input into the third ConvBNAct layer [Conv2d(8,16,kernel_size=2,stride=1,padding=0),BN(16),ReLU], and the feature map [B,16,H / 2,W / 2] is output to form the first branch;

[0191] The feature map [B, 16, H / 2+1, W / 2+1] output by the first ConvBNAct layer is input into the max pooling layer [MaxPool2d(kernel_size=2, stride=1, ceil_mode=True)] for dimensionality reduction and feature extraction, and the output feature map [B, 16, H / 2, W / 2] is formed to form the second branch;

[0192] The first branch and the second branch are concatenated at the Concat layer along the channel dimension to obtain a feature map of [B, 32, H / 2, W / 2].

[0193] The feature map [B,32,H / 2,W / 2] is input into the fourth ConvBNAct layer [Conv2d(32,16,kernel_size=3,stride=2,padding=1),BN(16),ReLU], and downsampled to output the feature map [B,16,H / 4,W / 4].

[0194] Then input the feature map [B,16,H / 4,W / 4] into the fifth ConvBNAct layer [Conv2d(16, 16, kernel_size=1, stride=1, padding=0), BN(16), ReLU], and output the feature map [B,16,H / 4,W / 4].

[0195] like Figure 3 The CSP-RCB module includes a twelfth Conv layer, a first split layer, a fifth Concat layer, a fourteenth Conv layer, and two RCB layers, namely the first RCB layer and the second RCB layer.

[0196] The specific implementation of the first CSP-RCB module is as follows:

[0197] Input the feature map [B,16,H / 4,W / 4] into the twelfth Conv layer [Conv2d(16,32,kernel_size=1, stride=1, padding=0, groups=1, bias=False), BatchNorm2d(32), SiLU] in the first CSP-RCB module, and output the feature map [B,32,H / 4,W / 4].

[0198] The feature map [B,32,H / 4,W / 4] is then input into the first split layer for splitting, the number of channels is halved, and two feature maps [B,16,H / 4,W / 4] are output.

[0199] Input any feature map of [B, 16, H / 4, W / 4] into the first RCB layer (the input and output remain unchanged in the RCB layer):

[0200] The feature map [B, 16, H / 4, W / 4] is input into the DW Conv layer, which is a 3×3 depthwise separable convolutional layer. The DW Conv layer performs deep feature extraction on the feature map and outputs the feature map [B, 16, H / 4, W / 4].

[0201] The feature map [B,16,H / 4,W / 4] output by the DW Conv layer is added to the feature map [B,16,H / 4,W / 4] before the input to the DW Conv layer to output the feature map [B,16,H / 4,W / 4].

[0202] The feature map [B, 16, H / 4, W / 4] obtained by summing is input into the LayerNorm layer to stabilize and normalize the training of the document element detection model, and the feature map [B, 16, H / 4, W / 4] is output.

[0203] The feature map [B, 16, H / 4, W / 4] output from the LayerNorm layer is input into the DilatedRepBlock layer, which outputs a feature map of [B, 16, H / 4, W / 4]. The DilatedRepBlock layer is a 7×7 depthwise separable convolution with dilation=1 and padding=3, followed by batch normalization (BN). The DilatedRepBlock layer includes four dilated branches: (k, r) (i.e., kernel size and dilation rate), which are (5, 1), (3, 1), (3, 2), and (3, 3) in sequence. Each branch is a depthwise separable convolution. During training, multiple branches participate in the training. During deployment, multiple branches can be merged into a single equivalent large kernel convolution through "switch_to_deploy" to facilitate inference acceleration.

[0204] The Dilated RepBlock layer not only expands the receptive field by inserting gaps between convolutional kernel elements, keeping the number of parameters and output resolution unchanged, enabling the network to integrate layout information over a wider range of the page and thus more accurately understand the semantic role of each local region in the global document structure; it also introduces reparameterization techniques: during the training phase, this layer adopts a multi-branch structure to learn richer feature representations; during the inference phase, these branches can be equivalently merged into a single convolutional layer, thereby improving inference speed without sacrificing performance.

[0205] The feature map [B, 16, H / 4, W / 4] output by the Dilated RepBlock layer is input into the SE layer, and the output feature map is [B, 16, H / 4, W / 4]. The SE layer enhances element attention: AdaptiveAvgPool2d(1) → Conv2d(in_channels, in_channels / / 8, 1) → GELU → Conv2d(in_channels / / 8, in_channels, 1) → Sigmoid, and then multiplies it element-wise with the input to perform channel recalibration. The SE layer adaptively recalibrates the feature response weights of each channel by modeling the interdependence between channels, so that the network can focus on the feature channels that are more important to the current document element classification task, thereby enhancing the discriminative power of the features.

[0206] The feature map [B,16,H / 4,W / 4] output from the SE layer is input into the thirteenth Conv layer for feature extraction, and the output feature map is [B,16,H / 4,W / 4].

[0207] The output of the thirteenth Conv layer is fused with the feature map [B,16,H / 4,W / 4] obtained by addition, and the feature map size remains unchanged, still being a feature map of [B,16,H / 4,W / 4]. The feature map of [B,16,H / 4,W / 4] is input into the second RCB layer, and the output is a feature map of [B,16,H / 4,W / 4]. The architecture of the second RCB layer is the same as that of the first RCB layer.

[0208] The two feature maps [B,16,H / 4,W / 4] output from the first split layer, and the feature maps [B,16,H / 4,W / 4] output from the first RCB layer and the second RCB layer are input into the fifth Concat layer for concatenation along the channel dimension, and the feature map [B,64,H / 4,W / 4] is output.

[0209] Input the feature map [B,64,H / 4,W / 4] output from the fifth Concat layer into the fourteenth Conv layer [Conv2d(64,64,kernel_size=1,stride=1,padding=0),BN(64),SiLU] in the first CSP-RCB module, and output the feature map [B,64,H / 4,W / 4].

[0210] The second CSP-RCB module differs from the first CSP-RCB module in all aspects of network layer operations and parameters, except for the following:

[0211] The parameters in the twelfth Conv layer are different: Conv2d(64, 64, kernel_size=1, stride=1, padding=0), BN(64), SiLU; output feature map [B, 64, H / 8, W / 8];

[0212] The parameters in the fourteenth Conv layer are different: Conv2d(128, 256, kernel_size=1, stride=1, padding=0), BN(256), SiLU; the output is a feature map of [B, 256, H / 8, W / 8], which is the input of P3;

[0213] The third CSP-RCB module differs from the first CSP-RCB module in all aspects of network layer operation and parameters, except for the following:

[0214] The parameters in the twelfth Conv layer are different: Conv2d(256, 128, kernel_size=1, stride=1), BN(128), SiLU; output feature map [B, 128, H / 16, W / 16];

[0215] The parameters in the fourteenth Conv layer are different: Conv2d(256, 512, kernel_size=1, stride=1), BN(512), SiLU; output feature map [B, 512, H / 16, W / 16];

[0216] The fourth CSP-RCB module differs from the first CSP-RCB module in the following aspects, while all other network layer operations and parameters are the same:

[0217] The parameters in the twelfth Conv layer are different: Conv2d(512, 128, kernel_size=1, stride=1), BN(128), SiLU; output feature map [B, 128, H / 16, W / 16];

[0218] The parameters in the fourteenth Conv layer are different: Conv2d(256, 512, kernel_size=1, stride=1), BN(512), SiLU; the output is a feature map of [B, 512, H / 16, W / 16], which is the feature map input to P4.

[0219] The fifth CSP-RCB module differs from the first CSP-RCB module in the following aspects, while all other network layer operations and parameters are the same:

[0220] The parameters in the twelfth Conv layer are different: Conv2d(512, 256, kernel_size=1, stride=1), BN(256), SiLU; output feature map [B, 256, H / 32, W / 32];

[0221] The parameters in the fourteenth Conv layer are different: Conv2d(512, 1024, kernel_size=1, stride=1), BN(1024), SiLU; the output is a feature map of [B, 1024, H / 32, W / 32], which is the feature map input to P5;

[0222] like Figure 4 The DIMB module includes a fifteenth Conv layer, a second split layer, a sixth Concat layer, an eighteenth Conv layer, and two DIDW modules, namely a first DIDW module and a second DIDW module.

[0223] The first DIMB module is implemented as follows:

[0224] The feature map [B,512,H / 16,W / 16] output from the first Concat layer is input into the fifteenth Conv layer [Conv2d(512,512,kernel_size=1,stride=1,padding=0,groups=1),BN(512),SiLU], and the output feature map is [B,512,H / 16,W / 16].

[0225] The feature map [B,512,H / 16,W / 16] output from the fifteenth Conv layer is input into the second split layer for splitting, and two feature maps [B,256,H / 16,W / 16] are output.

[0226] Input any one of the [B,256,H / 16,W / 16] feature maps output from the split layer into the first DIDW (Dynamic Inception DepthWise) module:

[0227] The feature maps [B,256,H / 16,W / 16] are input into the sixteenth Conv layer [Conv2d(256,64,kernel_size=1,stride=1),BN(64),SiLU] and the seventeenth Conv layer [Conv2d(256,64,kernel_size=1,stride=1),BN(64),SiLU], respectively, and both output the feature maps [B,64,H / 16,W / 16].

[0228] The feature map [B,64,H / 16,W / 16] output from the sixteenth Conv layer is input into the DIDWB (DynamicInceptionDWConv2d) module, which outputs a feature map [B,64,H / 16,W / 16].

[0229] After adding the DIDWB module to the feature map [B,64,H / 16,W / 16] output by the seventeenth Conv layer, the feature map is fed into the BatchNorm layer and the Silu layer in sequence, with the feature map size remaining unchanged, and the output feature map is [B,64,H / 16,W / 16].

[0230] The feature map [B,64,H / 16,W / 16] output by the first DIDW module is input into the second DIDW module, which outputs a feature map [B,64,H / 16,W / 16].

[0231] The second DIDW module differs from the first DIMB module in all aspects of network layer operations and parameters, except for the following:

[0232] The parameters in the sixteenth Conv layer are different: Conv2d(64, 64, kernel_size=1, stride=1), BN(64), SiLU; output feature map [B, 64, H / 16, W / 16];

[0233] The parameters in the seventeenth Conv layer are different: Conv2d(64, 64, kernel_size=1, stride=1), BN(64), SiLU; output feature map [B, 64, H / 16, W / 16];

[0234] The two feature maps [B,256,H / 16,W / 16] output from the second split layer, and the feature maps [B,64,H / 16,W / 16] output from the first and second DIDW modules are input into the sixth Concat layer for concatenation along the channel dimension to obtain the feature map [B,640,H / 16,W / 16].

[0235] The feature map [B, 640, H / 16, W / 16] is input into the 18th Conv layer [Conv2d(640, 256, kernel_size=1, stride=1, padding=0), BN(256), SiLU] to perform feature extraction and output the feature map [B, 256, H / 16, W / 16].

[0236] The network layer operations and parameters of the second, third, and fourth DIMB modules are the same as those of the first DIMB module, and will not be described again.

[0237] Among them, such as Figure 5 The specific implementation of the DIDWB module is as follows:

[0238] The feature map [B,C,H,W] is input into three depthwise separable convolutional layers, each outputting a feature map of [B,C,H,W]; where B is the batch size, C is the number of channels, H is the height, and W is the width. The three depthwise separable convolutional layers are a 1×K horizontal bar convolution, a 3×3 square convolution, and a K×1 vertical bar convolution. The 3×3 square convolution is responsible for capturing local, square feature patterns, suitable for processing block-shaped objects such as tables and images. The 1×K horizontal bar convolution is used to capture long-distance dependencies in the horizontal direction, which is extremely suitable for detecting horizontal bar elements such as chapter titles and formulas. The K×1 vertical bar convolution is used to capture vertical contextual information, which is crucial for understanding elements with vertical relationships (such as an image and its caption).

[0239] Dynamic convolution kernel weights: The feature map [B,C,H,W] is input into the AvgPool layer for average pooling. After outputting the feature map [B,C,1,1], a 1x1 convolution (Conv2d(C,3C,1)) is input to obtain a 3C-dimensional vector [B,3C,1,1]. Then, the shape is reshaped to [3,B,C,1,1]. The Softmax function is applied to normalize the first dimension (3) to obtain three sets of weights. Each set of weights corresponds to the output of a convolutional branch. Corresponding horizontal bar convolution branches (1×K). Corresponding to the square convolution branch (3×3), Corresponding vertical bar convolution branch (K×1);

[0240] After weighting and fusing the outputs of the three sets of weights with the outputs of the three depthwise separable convolutional layers respectively, and then passing them through BN and Silu activation respectively, the final fused feature map is obtained. The shape of the output remains unchanged and is still a feature map of [B,C,H,W].

[0241] In the network layers described in this invention, the Stem layer consists of a series of convolution, padding, pooling, and concatenation operations, used to achieve rapid downsampling and preliminary feature extraction of the input document image.

[0242] The Conv layer performs two-dimensional convolution operations, and its purpose varies depending on the configuration:

[0243] When the convolutional kernel size is 3×3, the stride is 2, and the padding is 1, the spatial size of the output feature map is exactly half the size of the input. This is used to achieve spatial downsampling of the feature map, aiming to reduce the feature map resolution, expand the receptive field, reduce the computational cost of subsequent layers, and align the feature map size. Examples include the first, second, third, tenth, and eleventh Conv layers.

[0244] When the convolution kernel size is 1×1 and the output size remains unchanged, it is used to achieve cross-channel information interaction and dimensionality reduction / expansion (projection) of feature maps. The purpose is to adjust the number of channels in the feature map to match the input requirements of subsequent layers, or to compress / expand features to facilitate feature fusion between different layers. Dimensionality reduction is seen in Conv 4 and 6. Cross-channel information interaction is seen in Conv 5 and 8. The AIFI layer is a Transformer-based encoder layer used to model long-distance dependencies of high-level semantic features. Its purpose is to utilize self-attention mechanisms to capture global contextual information between distant elements in the document image, solving the problem of limited receptive field in convolution operations, thereby more accurately understanding the overall layout and semantic structure of the document.

[0245] The Upsample layer is an upsampling operation used to improve the spatial resolution of the feature map. Its purpose is to restore the low-resolution high-level semantic feature map to a higher resolution so that it can be fused with shallow features with rich spatial details in the subsequent feature pyramid network (aligning the feature map size), thereby constructing multi-scale features that contain both strong semantic information and fine spatial location.

[0246] The Concat layer is a feature map concatenation operation used to fuse features from different paths or layers along the channel dimension. Its purpose is to merge features with different receptive fields and semantic levels (different resolutions), providing more comprehensive information for subsequent convolution or fusion modules. It is a key step in achieving multi-scale feature fusion.

[0247] The DIMB module is a dynamic multi-branch deep separable fusion module used to achieve refined fusion and re-extraction of multi-scale features after concatenation. It processes features through splitting and multi-branching DIDW modules (containing dynamically selected convolutional kernels of different shapes). The original branches retain the basic features, while the DIDW processing branches incorporate semantic information extracted by dynamic multi-scale convolution (convolutional kernels of different shapes). This allows the network to adaptively fuse features from different sources, enhancing the model's ability to represent diverse elements in document images (such as long titles and square images) after concatenation.

[0248] The CSP-RCB module is used to achieve efficient and powerful feature extraction. It splits the feature map into two paths: one is passed directly, and the other undergoes deep processing through multiple RCBs (including depthwise convolutions, layer normalization, dilated reparameterization blocks, and SE layers), before finally concatenating and fusing them. The feature map from the directly passed split path is concatenated with the feature map after deep RCB processing. The purpose of this concatenation is to fuse the original features with the features extracted by RCB along the channel dimension, thereby achieving information aggregation—preserving the unprocessed original information while introducing higher-order semantic features enhanced by RCB, enriching feature diversity and improving overall representation capabilities.

[0249] In this invention, the CSP-RCB module is a cross-stage locally reparameterizable context module, the RCB layer is a reparameterizable context module, the DIMB module is a dynamic multi-branch hybrid convolutional block, the DIDWB module is a dynamic depthwise separable weight block, the DIDW module is a dynamic depthwise separable weight, and the SE layer is a channel attention layer.

[0250] Given a document D consisting of P pages, represented as a sequence of page images:

[0251] ;

[0252] in, This represents the image on the Pth page;

[0253] Based on document D, a document element detection model Detect and categorize document elements on the page:

[0254] ;

[0255] in, , This indicates the page number on page p. The x and y coordinates of the top-left corner of the element's bounding box. This indicates the page number on page p. The x and y coordinates of the bottom right corner of the element's bounding box. These are category labels for document elements. The category labels vary depending on the dataset. For example, the category labels for Chinese scientific and technological literature include Title, Section title, Author and department, Text, Figure, Figure caption, Table, Table caption, Reference, Equation, and Footer. This represents the total number of document elements detected on page p.

[0256] Using OCR tools, i.e., optical character recognition engines Extract text content from text-type elements in a document:

[0257] ;

[0258] For non-text elements, their text content Set as tag type ;

[0259] The final output is a flattened collection containing all elements from all pages. :

[0260] ;

[0261] Each document element is represented by a tuple. This represents all of its attributes:

[0262] ;

[0263] Where p represents the document page number.

[0264] like Figure 6 A document hierarchy parsing model is constructed, which includes an embedding layer, an encoder, an interaction layer, and a gating fusion layer.

[0265] In step S2, the embedding layer is specifically as follows:

[0266] For page p of the document, divide according to coordinates Sort the document elements by ascending order of their y-coordinates (vertical coordinates), and by ascending order of their x-coordinates (horizontal coordinates) if they are the same y-coordinate, to obtain an ordered list. ;

[0267] For document elements Use a pre-trained word segmenter to Segmented into a sequence of subwords, non-text elements use their category information. As text fill Get the token list ,in This is the length of the tokens in this element; all tokens within the same element share the element's bounding box. ; Arrange the obtained list of tokens belonging to each document element within the page into an ordered list Arrange them in order, then add [CLS] and [SEP] to get That is, the input sequence of the layoutlmv3 model. ,in, This represents the total number of tokens on page p.

[0268] Here, the embedding of a token in a document element is represented as follows: , It integrates text, location, visual, page, and type information, and the specific calculation is as follows:

[0269] ;

[0270] in, This represents the input embedding representation of the k-th token in the p-th document element of the page. For text embedding, the element text is converted into a token ID sequence using a tokenizer, and then the token is retrieved using the layoutlmv3 token embedding matrix. For 1D positional embedding, it is obtained by positional encoding based on the position index of each token in the input sequence; For 2D position embedding, the layout embedding layer is obtained by assigning a corresponding bounding box to each token. For visual embedding, the page image is divided into fixed-size patches, which are then flattened and obtained through linear projection. Embed the page number for page P; Embedded for element type;

[0271] The page number embedding is calculated as follows:

[0272] Given a page number to which a token belongs Using sine-cosine position coding function Will After being transformed into a continuous vector, it is then passed through a linear projection layer. The sinusoidal encoding is mapped to a space aligned with the hidden layer dimensions of the document hierarchy parsing model, as follows:

[0273] ;

[0274] The element type embedding is calculated by assigning a predefined category label to each document element. Maintain document element type embedded lookup table ,in, The total number of types, For the embedding dimension; for type is The element type embedding for each token within a document element is obtained through a direct table lookup. ;

[0275] The embedding layer design brings the following benefits to multi-page document structure parsing: Page order awareness: Page number embedding enables the model to distinguish information from different pages, providing direct sequence clues for understanding the relationships between cross-page elements (such as determining whether two text blocks on different pages belong to the same chapter). Semantic function differentiation: Element type embedding provides the model with prior knowledge about the high-level function of the element to which each token belongs, greatly assisting the model in distinguishing document logical hierarchy (such as heading system) and functional areas (such as charts, body text, and annotations). The embedding layer does not change the core model architecture, retaining all its original advantages, while significantly enhancing the representation capabilities in multi-page, structured scenarios.

[0276] The physical structure of a document is naturally organized in units of pages. Each page contains multiple document elements. The core motivation for page-by-page encoding is to bypass the inherent sequence length limitation of the model and transform the encoding problem of extremely long documents into a parallel processing problem of multiple standard-length pages.

[0277] The input sequence constructed for each page The data is fed into the layoutlmv3 encoder, enabling each page to be encoded and learn industry context, regardless of the total document length.

[0278] In the encoder, the input sequence After inputting the layoutlmv3 encoder, we obtain the context-aware hidden state of each token:

[0279] ;

[0280] While the token-level representation output page by page is rich in detail, it is not an ideal unit for parsing document structure. The logical structure of a document consists of layout elements such as headings, paragraphs, lists, and charts, each of which may contain multiple tokens. Therefore, we need to "represent" the token-level features onto their respective layout elements.

[0281] By aggregating features of document elements:

[0282] Split the p-th page of the document into The document element, let the first Page 1 The set of token indexes contained in each document element is The token representation within the document element is aggregated using average pooling to obtain a compact vector of the document element. The specific calculations are as follows:

[0283] ;

[0284] Iterate through all the pages in the document to get There are 1 document element, P is the total number of pages; based on the compact vector of document elements. By arranging the document elements into a sequence based on the page numbers and vertical coordinates within the page in the original document, a global element representation sequence is constructed. The specific calculations are as follows:

[0285] ;

[0286] in, For the i-th globally sorted document element, For embedded dimensions; the sorting rules are as follows: first, sort by page number p in ascending order; for elements within the same page, sort by the coordinates of the document elements within the page, that is, first sort by the vertical coordinate y in ascending order, and if y is the same, then sort by the horizontal coordinate x in ascending order.

[0287] Through a two-stage strategy of "page-by-page encoding" and "element aggregation", the model not only makes full use of the powerful capabilities of the page-level pre-trained model, but also generates a compact and semantic representation suitable for reasoning about the hierarchical structure of documents, laying a solid foundation for the entire parsing process.

[0288] In step S2, such as Figure 7 The interaction layer is:

[0289] Given a global element representing a sequence Candidate chapter title elements are filtered based on predefined element type tags. ; ,in, Number of candidate chapter titles For the embedded dimension;

[0290] Calculate the semantic relationships between candidate chapter title elements, mapping them to the query space Q, key space K, and value space V. The specific calculation is as follows:

[0291] ;

[0292] ;

[0293] ;

[0294] in, These are learnable parameters;

[0295] Based on the query space Q and the key space K, the mutual attention among candidate chapter titles is calculated. The specific calculation formula is as follows:

[0296] ;

[0297] Where T is the transpose. For projection dimensions;

[0298] Based on the mutual attention level A, the attention output is:

[0299] ;

[0300] Projection through attention output By performing residual connections and layer normalization, we obtain a context-enhanced chapter element representation. Specifically:

[0301] ;

[0302] in, This is the normalization function;

[0303] Sum the values ​​in each column of mutual attention A to obtain the importance score for each candidate chapter title. :

[0304] ;

[0305] The total important score for all candidate chapter titles is: ;

[0306] Normalize the total importance score of all candidate chapter titles to obtain the importance distribution. :

[0307] ;

[0308] Based on α, the Top-K candidate nodes with the highest scores are selected as chapter boundary nodes; the chapter boundary nodes divide the document into K+1 local semantic windows; each local semantic window defined by the chapter boundary node... A primary semantic unit of the corresponding document;

[0309] For local semantic windows Document elements within are represented as The enhanced representation is obtained through an L-layer Transformer encoder. The specific calculations are as follows:

[0310] ;

[0311] in, d represents the local semantic window, and d represents the embedding dimension;

[0312] After all local semantic window interactions are completed, the enhanced document element representation is obtained as follows: :

[0313] ;

[0314] Where N is the total number of document elements and d is the embedding dimension.

[0315] In step S2, such as Figure 8 The gated fusion layer is:

[0316] Based on enhanced document elements Construct joint representation vector Where i, j represent the chapter element pair (i, j); Inputting the data into a multilayer perceptron classifier (MLP) yields a global relation prediction head, i.e., a relation matrix. The specific calculation formula is as follows:

[0317] ;

[0318] MLP stands for Multilayer Perceptron; The dimension is r, where r is the relation type;

[0319] A dual-branch gating fusion mechanism is designed, consisting of local branches and chapter branches. The local branches utilize global relationships to predict the prediction results for head-to-chapter element pairs. The specific calculation is as follows:

[0320] ;

[0321] in, That is, the relation matrix 'r' represents the relation type. For the index set of candidate chapter title elements , Number of candidate chapter titles;

[0322] Chapter branches are represented by context-enhanced chapter elements. Predict the relation by constructing a joint representation vector for each pair of chapter elements (i,j). Input it into a multilayer perceptron The prediction results of chapter branches are obtained. ;

[0323] Gated networks dynamically generate fusion weight vectors for each relation type based on the context of local and chapter branches. :

[0324] ;

[0325] in, and For learnable parameters, For activation functions; They are respectively Features of the i-th and j-th chapter elements; They are respectively Features of the i-th and j-th chapter elements;

[0326] The final inter-chapter relationship prediction is obtained through weighted fusion:

[0327] ;

[0328] Will Write back the relation matrix of all elements Calculate the probability of the final relationship:

[0329] ;

[0330] Employing multi-label binary cross-entropy loss The training process is performed, and a mask is applied to avoid invalid pairings, as follows:

[0331] ;

[0332] in, For real labels, i and j represent chapter element pairs (i, j), r is the relation type, and R is the total number of relation types; The function is the Sigmoid function, where N is the total number of document elements;

[0333] Inputting document elements into the document hierarchy analysis model yields the final relation probabilities. Determine the direct relationships between elements based on the final relationship probability. In the calculation It is obtained based on MLP, with dimension r, which is the relation type. The relation type is divided into parent-child relation and sequential relation. When there are two relation types (both parent-child relation and sequential relation exist), r=2; when there is only one relation type, r=1. The relation type is labeled by the dataset itself. For example, Chinese scientific and technological literature is labeled with parent-child relation and sequential relation, while the Dochienet dataset is only labeled with parent-child relation.

[0334] The Chinese Scientific and Technological Literature Dataset is a standard dataset designed for deep learning-based document structure parsing tasks. It contains 315 academic papers covering multiple fields such as physics, mathematics, and computer science.

[0335] Dochienet is a dataset specifically designed for the task of annotating the hierarchical structure of multi-page documents, providing rich information on layout elements and their relationships.

[0336] It contains two matrices, one containing the predicted values ​​and the other observing the parent-child relationship. When evaluating the relationship using F1, the predicted values ​​in the matrices are greater than 0, and the probabilities are obtained after passing through the sigmoid activation function. If the matrix value is greater than 0.5, the two elements are considered to be related; otherwise, if the matrix value is less than 0, the probability obtained after the sigmoid activation function is less than 0.5, and the two elements are considered to be unrelated; thus, the complete document element relationships are identified.

[0337] Among them, the father-son relationship Defines the hierarchical nesting structure of elements; relationships Represents element It is an element The parent node, where It is a flattened collection that contains all elements of all pages.

[0338] Order relation Defines the reading order of elements; relationships This indicates that during the reading process, elements Following the elements after.

[0339] An ordered tree can be constructed using parent-child relationships and sequence relationships. Structured tree representation It achieves deep semantic understanding and reconstruction of original document image sequences;

[0340] The ordered tree is constructed as follows:

[0341] (1) Establish a hierarchical framework based on the parent-child relationship

[0342] First, construct the basic hierarchical structure of the tree using parent-child relationships; for each pair of relationships... Create a sub-node in the tree Pointing to node The directed edges, where become The parent node; by traversing the collection All relations in the set can flatten the initial set of elements. Organize it into a hierarchical structure.

[0343] To form a unified tree that represents the entire document, a virtual root node is introduced. This node does not correspond to any specific document element. Its function is to aggregate all top-level elements without parent nodes (such as the document's main title, author, etc.) into its child nodes, thereby ensuring that the final output is a single, complete tree.

[0344] (2) Determine the node order based on the sequence relationship.

[0345] Based on After establishing the hierarchical tree structure, the correct reading order needs to be assigned to the list of child nodes under each parent node using sequential relationships; sequential relationships This indicates that during the reading process, elements Following the elements after;

[0346] In the context of tree construction, this relationship only applies to elements. and The relative order of sibling nodes is only used to determine their relative order when they share the same direct parent node in an already constructed tree. Specifically, this relationship indicates... It should be determined as The next sibling node. By systematically traversing and organizing all the order relationships acting on the same parent node, a definite, ordered sequence that conforms to human reading habits can be defined for the set of child nodes of each internal node (i.e., the node that has child nodes) in the tree.

[0347] Meanwhile, this invention provides:

[0348] A server includes a processor and a memory, the memory storing at least one program that is loaded and executed by the processor to implement the above-described method for document content recognition and main text section tree structure analysis.

[0349] A computer-readable storage medium storing at least one program, which is loaded and executed by a processor to implement the above-described method for document content recognition and main text section tree structure analysis.

[0350] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for document content recognition and main text section tree structure analysis, characterized in that, Includes the following steps: S1. Collect the document and convert it into an image format. Input the document image into the document element detection model to obtain the document elements. The document element detection model comprises an input layer, an encoding layer, and a decoding layer. Five cross-stage locally reparameterizable context modules (CSP-RCB) are constructed in the input layer. The first and third CSP-RCB modules enhance the feature extraction capability of the document content. The features output by the second, fourth, and fifth CSP-RCB modules in the input layer are defined as shallow, mid-level, and deep features, respectively, and then input into the encoding layer to perceive the document's detailed information and high-level semantics. A dynamic multi-branch hybrid convolutional block (DIMB) is constructed in the encoding layer to achieve refined fusion and re-extraction of multi-scale features of the document. The output of the encoding layer is input into the decoding layer to obtain the document elements. The CSP-RCB module includes a twelfth Conv layer, a first split layer, a fifth Concat layer, a fourteenth Conv layer, and two RCB layers, namely the first RCB layer and the second RCB layer. The CSP-RCB module workflow is as follows: Input feature map A into the twelfth Conv layer, adjust the number of channels of the feature map to match the input requirements of subsequent layers, and output feature map B; Feature map B is input into the first split layer and split into two feature maps C and D; Any feature map obtained after splitting the first split layer is input into the first RCB layer for deep feature extraction and enhancement of high-order semantic features, and a feature map E is output. Input feature map E into the second RCB layer and output feature map F; Feature maps C, D, E, and F are input together into the fifth Concat layer for concatenation. The original features are fused with the features extracted by the RCB layer in the channel dimension to achieve information aggregation, improve the overall representation ability, and output feature map G. The feature map G is input into the fourteenth Conv layer to perform cross-channel information interaction of the feature maps and output the feature map H as the output of the CSP-RCB module. The network layer architecture and parameters of the first RCB layer and the second RCB layer are the same, and the input and output feature map sizes are unchanged; the RCB layer includes the DW Conv layer, LayerNorm layer, Dilated RepBlock layer, SE layer and the thirteenth Conv layer; The specific workflow of the RCB layer is as follows: The feature map a is input into the DW Conv layer, and a depthwise separable convolution is performed to extract deep features from the feature map, outputting feature map b. Add feature map a to feature map b to output feature map c; Input feature map c into LayerNorm to stabilize and normalize the document element detection model during training, and output feature map d; The feature map d is input into the Dilated RepBlock layer. While keeping the number of parameters and the output resolution unchanged, the receptive field is expanded and layout information over a larger range of the document page is integrated, so as to accurately understand the semantic role of each local region in the global document structure and output the feature map e. The feature map e is input into the SE layer to model the interdependencies between channels, adaptively recalibrate the feature response weights of each channel, and output the feature map f. The feature map f is input into the thirteenth Conv layer to perform cross-channel information interaction of the feature maps and output the feature map g. Add feature map g to feature map c to output feature map h; The DIMB module includes a fifteenth Conv layer, a second split layer, a sixth Concat layer, an eighteenth Conv layer, and two DIDW modules, namely the first DIDW module and the second DIDW module. The specific workflow of the DIMB module is as follows: Feature map I is input into the fifteenth Conv layer to perform cross-channel information interaction of feature maps and output feature map J. The feature map J is input into the second split layer for splitting, and two feature maps K and L are output. Any feature map obtained from the second split layer is input into the first DIDW module, and semantic information extracted by dynamic multi-scale convolution is incorporated to adaptively fuse features from different sources and output feature map M. Input the feature map M into the second DIDW module to extract multi-scale features and output the feature map N; Feature maps K, L, M, and N are input together into the sixth Concat layer for concatenation along the channel dimension, which enhances the document element detection model's ability to represent diverse elements in document images and outputs feature map O. The feature map O is input into the eighteenth Conv layer for dimensionality reduction, and the output feature map P is obtained. The DIDW module includes a sixteenth Conv layer, a seventeenth Conv layer, a DIDWB module, a BatchNorm layer, and a Silu layer; The specific workflow of the DIDW module is as follows: Input feature map i into the sixteenth Conv layer for dimensionality reduction, and output feature map j; Input feature map j into the DIDWB module to capture local and different features, perform weighted fusion of features, and output feature map k; Input feature map i into the seventeenth Conv layer for dimensionality reduction, and output feature map l; Add feature map k to feature map l to output feature map m; The feature map m is sequentially fed into the BatchNorm layer and the Silu layer to stabilize the training of the document element detection model, and the feature map n is output. The specific process of the DIDWB module is as follows: The feature map [B,C,H,W] is input into three depthwise separable convolutional layers, each outputting a feature map of [B,C,H,W]; where B is the batch size, C is the number of channels, H is the height, and W is the width; the three depthwise separable convolutional layers are horizontal bar convolution, square convolution, and vertical bar convolution; the square convolution is responsible for capturing local, square feature patterns; the horizontal bar convolution captures long-range dependencies in the horizontal direction; and the vertical bar convolution captures contextual information in the vertical direction. The feature map [B,C,H,W] is input into the AvgPool layer for average pooling, outputting a feature map [B,C,1,1]. This output feature map is then fed into a 1x1 convolution to obtain a 3C-dimensional vector [B,3C,1,1]. The shape is then reshaped to [3,B,C,1,1]. Softmax function is applied to the first dimension for normalization, resulting in three sets of weights. Each set of weights corresponds to the output of a convolutional branch. Corresponding to the horizontal bar convolution branch, Corresponding to the square convolution branch, Corresponding vertical bar convolution branch; After weighting and fusing the outputs of the three sets of weights with the outputs of the three depthwise separable convolutional layers respectively, the final fused feature map is obtained by passing it through BatchNorm and Silu activation respectively. The shape of the output remains unchanged and is still a feature map of [B,C,H,W]. S2. Input the document elements into the document hierarchy parsing model to obtain the final relationship probability of the document elements. Based on the final relationship probability, obtain the tree hierarchy structure and order relationship between the document elements, thereby identifying the complete document. The document hierarchy parsing model includes an embedding layer, an encoder, an interaction layer, and a gated fusion layer. The embedding layer adds page number embedding and element type embedding to obtain the input sequence. After the input sequence is encoded by the encoder, the interaction layer performs context enhancement on chapter elements and local window document elements. Based on the context enhancement of chapter elements and local window document elements, the gated fusion layer fuses the prediction results of local window document elements and chapter elements to obtain the final relationship probability.

2. The method for document content recognition and main text section tree structure analysis according to claim 1, characterized in that, In step S1, the document elements are specifically as follows: Given a document D consisting of P pages, represented as a sequence of page images: ； in, This represents the image on the Pth page; Based on document D, a document element detection model Detect and categorize document elements on the page: ； in, , This indicates the page number on page p. The x and y coordinates of the top-left corner of the element's bounding box. This indicates the page number on page p. The x and y coordinates of the bottom right corner of the element's bounding box. Category tags for document elements, including Title, Text, Figure, and Table; This represents the total number of document elements detected on page p. Through optical character recognition engine Extract text content from text-type elements in a document: ； For non-text elements, their text content Set as tag type ; The final output is a flattened collection containing all elements from all pages. : ； Each document element is represented by a tuple. This represents all of its attributes: ； Where p represents the document page number.

3. The method for document content recognition and main text section tree structure analysis according to claim 1, characterized in that, In step S2, the embedding layer is specifically as follows: For page p of the document, divide according to coordinates Sort the document elements by ascending order of their y-coordinates (vertical coordinates), and by ascending order of their x-coordinates (horizontal coordinates) if they are the same y-coordinate, to obtain an ordered list. ; For document elements Using a pre-trained word segmenter Segmented into a sequence of subwords, non-text elements use their category information. As text fill Get the token list ,in This is the length of the tokens in this element; all tokens within the same element share the element's bounding box. ; Arrange the obtained list of tokens belonging to each document element within the page into an ordered list Arrange them in order, then add [CLS] and [SEP] to get That is, the input sequence of the layoutlmv3 model. ,in, This represents the total number of tokens on page p. Here, the embedding of a token in a document element is represented as follows: , It integrates text, location, visual, page, and type information, and the specific calculation is as follows: ； in, This represents the input embedding representation of the k-th token in the p-th document element of the page. For text embedding, the element text is converted into a token ID sequence using a tokenizer, and then the token is retrieved using the layoutlmv3 token embedding matrix. For 1D positional embedding, it is obtained by positional encoding based on the position index of each token in the input sequence; For 2D position embedding, the layout embedding layer is obtained by assigning a corresponding bounding box to each token. For visual embedding, the page image is divided into fixed-size patches, which are then flattened and obtained through linear projection. Embed the page number for page P; Embedded for element type; The page number embedding is calculated as follows: Given a page number to which a token belongs Using the sine-cosine position coding function Will After being transformed into a continuous vector, it is then passed through a linear projection layer. The sinusoidal encoding is mapped to a space aligned with the hidden layer dimensions of the document hierarchy parsing model, as follows: ； The element type embedding is calculated by assigning a predefined category label to each document element. Maintain document element type embedded lookup table ,in, The total number of types, For the embedding dimension; for type is The element type embedding for each token within a document element is obtained through a direct table lookup. .

4. The method for document content recognition and main text section tree structure analysis according to claim 1, characterized in that, In step S2, the encoder encodes the document page by page, inputting a sequence. After inputting the layoutlmv3 encoder, we obtain the context-aware hidden state of each token: ； Feature aggregation of document elements: Split the p-th page of the document into The document element, let the first Page 1 The set of token indexes contained in each document element is The token representation within the document element is aggregated using average pooling to obtain a compact vector of the document element. The specific calculations are as follows: ； Iterate through all the pages in the document to get There are 1 document element, P is the total number of pages; based on the compact vector of document elements. By arranging the document elements into a sequence based on the page numbers and vertical coordinates within the page in the original document, a global element representation sequence is constructed. The specific calculations are as follows: ； in, For the i-th globally sorted document element, For embedded dimensions; the sorting rules are as follows: first, sort by page number p in ascending order; for elements within the same page, sort by the coordinates of the document elements within the page, that is, first sort by the vertical coordinate y in ascending order, and if y is the same, then sort by the horizontal coordinate x in ascending order.

5. The method for document content recognition and main text section tree structure analysis according to claim 1, characterized in that, In step S2, the interaction layer is: Given a global element representing a sequence Candidate chapter title elements are filtered based on predefined element type tags. ; ,in, Number of candidate chapter titles For the embedded dimension; Calculate the semantic relationships between candidate chapter title elements, mapping them to the query space Q, key space K, and value space V. The specific calculation is as follows: ； ； ； in, These are learnable parameters; Based on the query space Q and the key space K, the mutual attention among candidate chapter titles is calculated. The specific calculation formula is as follows: ； Where T is the transpose. For projection dimensions; Based on the mutual attention level A, the attention output is: ； Projection through attention output By performing residual connections and layer normalization, we obtain a context-enhanced chapter element representation. Specifically: ； in, This is the normalization function; Sum the values ​​in each column of mutual attention A to obtain the importance score for each candidate chapter title. : ； The total important score for all candidate chapter titles is: ; Normalize the total importance scores of all candidate chapter titles to obtain the importance distribution. : ； Based on α, the Top-K candidate nodes with the highest scores are selected as chapter boundary nodes; the chapter boundary nodes divide the document into K+1 local semantic windows; each local semantic window defined by the chapter boundary node... A primary semantic unit of the corresponding document; For local semantic windows Document elements within are represented as The enhanced representation is obtained through an L-layer Transformer encoder. The specific calculations are as follows: ； in, d represents the local semantic window, and d represents the embedding dimension; After all local semantic window interactions are completed, the enhanced document element representation is obtained as follows: : ； Where N is the total number of document elements and d is the embedding dimension.

6. The method for document content recognition and main text section tree structure analysis according to claim 1, characterized in that, In step S2, the gated fusion layer is: Based on enhanced document elements Construct joint representation vector Where i, j represent the chapter element pair (i, j); Inputting the data into a multilayer perceptron classifier (MLP) yields a global relation prediction head, i.e., a relation matrix. The specific calculation formula is as follows: ； MLP stands for Multilayer Perceptron; The dimension is r, where r is the relation type; A dual-branch gating fusion mechanism is designed, consisting of local branches and chapter branches. The local branches utilize global relationships to predict the prediction results for head-to-chapter element pairs. The specific calculation is as follows: ； in, That is, the relation matrix 'r' represents the relation type. For the index set of candidate chapter title elements , Number of candidate chapter titles; Chapter branches are represented by context-enhanced chapter elements. Predict the relation by constructing a joint representation vector for each pair of chapter elements (i,j). Input it into the multilayer perceptron The prediction results of chapter branches are obtained. ; Gated networks dynamically generate fusion weight vectors for each relation type based on the context of local and chapter branches. : ； in, and For learnable parameters, For activation functions; They are respectively Features of the i-th and j-th chapter elements; They are respectively Features of the i-th and j-th chapter elements; The final inter-chapter relationship prediction is obtained through weighted fusion: ； Will Write back the relation matrix of all elements Calculate the probability of the final relationship: ； Employing multi-label binary cross-entropy loss The training process is performed, and a mask is applied to avoid invalid pairings, as follows: ； in, For real labels, i and j represent chapter element pairs (i, j), r is the relation type, and R is the total number of relation types; Here, is the Sigmoid function, and N is the total number of document elements.

Images