A multi-modal fusion complex layout document structuring generation method and system

Abstract

The application belongs to the technical field of document processing, and particularly relates to a multi-modal fusion complex layout document structured generation method and system, which comprises the following steps: parsing a document to obtain a document object unit set, extracting a two-dimensional coordinate position information vector and a visual area feature vector, additionally extracting a content semantic information vector and a layout attribute vector of a text block. A pre-training model is inputted, a feature space correlation weighted attention mechanism is used in an encoder to generate a spatial relationship distribution parameter and inject a spatial bias term. A decoder inputs a virtual root node to generate a document object unit parent node candidate probability distribution, a global optimal tree-shaped logical structure is obtained by using a graph search algorithm, and the global optimal tree-shaped logical structure is converted into a structured data stream. By fusing multi-modal features and spatial relationship bias and combining a global optimal tree-shaped graph solving mechanism, the application realizes accurate and complete generation of a complex document logical structure.

Description

Technical Field

[0001] This invention relates to the field of document processing technology. More specifically, this invention relates to a method and system for generating complex layout documents using multimodal fusion. Background Technology

[0002] In fields such as intelligent question answering, information extraction, knowledge graph construction, and data processing, the prerequisite for in-depth utilization of unstructured and semi-structured document content is the structured parsing of documents. The key to structured parsing lies in identifying the logical hierarchical relationships within the document and transforming these relationships from a purely visual presentation into a machine-readable tree or graphical structure. Since the logical structure of a document depends not only on the semantic information of the text content but also implicitly conveyed through visual cues, effectively integrating multimodal information such as text semantics, layout attributes, and spatial arrangement to accurately parse the logical structure of complex document layouts has become a core technical challenge in the field of document intelligence.

[0003] Currently, while pre-trained models based on the Transformer architecture can detect spatial relationships between document objects using two-dimensional positional encoding and learn dependencies between different objects through self-attention mechanisms, they still face technical bottlenecks while improving recognition performance. On the one hand, existing attention mechanisms struggle to effectively represent the essential differences in spatial relationships between different types of document objects; on the other hand, existing models fail to establish a deep association between the layout attributes, semantic features, and spatial influence of objects, which directly limits the depth of the model's understanding of complex layouts.

[0004] During the logical structure generation stage, the lack of such feature associations further impacts prediction accuracy. Many existing methods employ autoregressive decoding to predict the parent nodes of document objects one by one, but this sequential decision-making process is highly susceptible to error accumulation. Early prediction errors severely interfere with the logical judgments of subsequent nodes, resulting in a generated structure that only achieves local optima. Due to the lack of global coordination regarding the consistency of the overall document structure, existing decoding mechanisms cannot guarantee the generation of a unique and globally optimal tree-like logical structure. Summary of the Invention

[0005] To address the shortcomings of existing technologies in effectively establishing the relationship between the layout attribute vector and content semantic information vector of document objects and spatial influence, and in the lack of global coordination for the overall structural consistency of documents during the logical structure generation stage, this invention provides a multimodal fusion method and system for generating complex layout documents.

[0006] In a first aspect, the present invention provides a method for structurally generating complex layout documents using multimodal fusion, comprising: parsing a document to obtain a set of document object units; extracting a two-dimensional coordinate position information vector and a visual feature vector for each document object unit, and extracting a content semantic information vector and a layout attribute vector when the document object unit is a text block; inputting the extracted features into a pre-trained model containing an encoder and a decoder; in the encoder, employing a feature space correlation weighted attention mechanism to generate a parameter set of spatial relationship distribution based on the corresponding features for each source document object unit; calculating a spatial relationship bias term based on the parameter set and injecting it into the calculation. From the original attention score matrix of self-attention; the embedding vector is input into the decoder as a virtual root node. For each document object unit, based on its decoding output state and the decoding output states of other units including the virtual root node, a candidate probability distribution with other units or the virtual root node as parent nodes is generated; a weighted directed graph is constructed according to the parent node candidate probability distribution among all document object units; a graph search algorithm is used to determine the unique globally optimal tree logical structure with the virtual root node as the root in the weighted directed graph; based on the globally optimal tree logical structure, the document object units and their corresponding logical hierarchy relationships are converted into a structured data stream in a preset format.

[0007] By adopting the above technical solution, in the feature encoding stage, the present invention uses a feature space correlation weighted attention mechanism to dynamically generate two-dimensional Gaussian distribution parameters based on the layout and semantic features of the unit itself and injects spatial relationship bias terms, thereby detecting the differences in spatial correlation between different types of units and improving the ability to perceive the internal connections of complex layouts. In the logic decoding stage, by inputting virtual root nodes and converting the tree structure generation into the construction of a weighted directed graph, the maximum generating tree graph algorithm is used for global solution, avoiding the error accumulation and structural breakage problems caused by local decision errors in serial autoregressive decoding, and ensuring that a unique and globally consistent optimal tree logic structure can be found from all possible parent-child relationships.

[0008] Preferably, the step of extracting a two-dimensional coordinate position information vector and a visual feature vector for each document object unit, and extracting a content semantic information vector and a layout attribute vector when the document object unit is a text block, includes: for each document object unit, extracting the vertex coordinates of its bounding box as a two-dimensional coordinate position information vector; inputting the pixel region of each document object unit into a pre-trained visual feature extraction network to extract the visual feature vector; for text block units, using a pre-trained language model to convert the text content into the content semantic information vector; and for text block units, extracting the font size value, the mapping value corresponding to the character weight, and the ratio of line spacing to font size, which together constitute the layout attribute vector.

[0009] By adopting the above technical solutions, this invention introduces a pre-trained visual feature extraction network to extract visual region feature vectors, providing underlying visual supplementation for distinguishing non-text elements such as images and tables, as well as detecting subtle differences in layout; by introducing a pre-trained language model to extract high-dimensional content semantic information vectors, and combining them with precisely calculated font, character weight mapping, and line spacing ratio to construct layout attribute vectors, it achieves multi-dimensional and high-precision stripping of the underlying information of complex layouts, providing a high-quality and highly representative data foundation for the subsequent encoder to calculate the correlation of feature space.

[0010] Preferably, the parameter set for generating the spatial relationship distribution includes: concatenating the two-dimensional coordinate position information vector, visual region feature vector, layout attribute vector, and content semantic information vector of the source document object unit to obtain a fused feature vector; when the source document object unit is a non-text block, a preset placeholder vector is used instead of the layout attribute vector and content semantic information vector; the fused feature vector is input into a three-layer multilayer perceptron, and the multilayer perceptron outputs five parameter values: which are respectively the first mean of the two-dimensional Gaussian distribution, which is the specific form of the spatial relationship distribution. Second mean First standard deviation Second standard deviation and correlation coefficient ; where the first standard deviation Second standard deviation Apply the Softplus activation function to the correlation coefficient By applying the hyperbolic tangent activation function, the above five parameters uniquely determine a two-dimensional Gaussian distribution.

[0011] By adopting the above technical solution, this invention achieves deep fusion and nonlinear mapping of heterogeneous features by splicing multimodal feature vectors and inputting them into a three-layer multilayer perceptron. Furthermore, the design of placeholder vectors for non-text blocks enhances the stability of the model. More importantly, by applying the Softplus activation function to the standard deviation and the hyperbolic tangent activation function to the correlation coefficient, the generated standard deviation is always positive and the correlation coefficient is in the range of [-1,1], ensuring the rationality of the two-dimensional Gaussian distribution generated by the model in terms of spatial geometry.

[0012] Preferably, the step of calculating the spatial relationship bias term based on the parameter set and injecting it into the original attention score matrix for calculating self-attention includes: for any source document object unit i and target document object unit j, calculating the center point coordinate offset of target document object unit j relative to source document object unit i; substituting the center point coordinate offset into the two-dimensional Gaussian distribution generated by source document object unit i, and calculating the value of the two-dimensional Gaussian distribution after removing the dimensionless exponent of the normalization coefficient, as the spatial relationship bias term. The spatial relationship bias terms between all document object pairs are used to form the bias matrix B. When calculating the self-attention score, the original attention score matrix A and the bias matrix B are added element-wise to obtain the new attention score matrix. Then apply the new attention score matrix Each row is processed using a normalized exponential function.

[0013] By adopting the above technical solution, this invention calculates the degree of dependence between two specific document objects under specific spatial layout semantics by substituting the center point coordinate offset of the target document object unit relative to the source document object unit into a dynamically generated two-dimensional Gaussian distribution formula. The bias matrix is ​​then added element-wise to the original attention score matrix and processed by a normalized exponential function, so that the attention weight no longer depends solely on the literal similarity of the content, but can adaptively perceive the spatial radiation range and direction determined by the layout and semantics of the source document object, thereby improving the accuracy of the model in capturing long-distance spatial dependencies.

[0014] Preferably, generating the candidate probability distribution with other units or virtual root nodes as parent nodes includes: inputting the hidden state vector of each document object unit output by the decoder, and the learnable embedding vector representing the virtual root node, into a bilinear classifier; for any document object unit that is a target document object unit, the bilinear classifier receives its corresponding hidden state vector, and sequentially calculates it with the learnable embedding vector corresponding to the virtual root node and the hidden state vectors corresponding to all other units, to output a set of probability values ​​for the target document object unit; wherein each probability value in the set of probability values ​​corresponds to the possibility of one of the other units or the virtual root node being a parent node, and the set of probability values ​​together constitute the parent node candidate probability distribution.

[0015] By adopting the above technical solution, this invention utilizes a bilinear classifier to receive the hidden state vector and the learnable embedding vector representing the virtual root node for cross-calculation, which can accurately characterize the scalar score of the potential parent-child logical relationship between any two nodes. By transforming this score into a probability distribution, it can not only evaluate all possible global dependencies in a parallel and efficient manner, but also smoothly integrate the virtual root node into the probability evaluation system, providing accurate conditional probability numerical basis for the subsequent construction of directed graphs.

[0016] Preferably, the step of constructing a weighted directed graph based on the candidate probability distribution of parent nodes among all document object units includes: setting the set of document object units as a first set U, and the virtual root node as... Then the set of vertices in the weighted directed graph is the first set U and the virtual root node. The union of the two vertices in a weighted directed graph; for any two vertices in the graph, i.e., the source node. and target node The source node Belongs to the set of vertices and target node For elements belonging to the first set U, and whose index i is not equal to j, construct a path from the source node. Point to target node The directed edges; the weight of the directed edges is set to the weight of the source node predicted by the decoder. The target node The conditional probability value of the parent node.

[0017] Preferably, the step of using a graph search algorithm to determine the unique globally optimal tree logical structure rooted at the virtual root node in a weighted directed graph includes: using the Julius Edmonds algorithm, for each node other than the virtual root node, selecting and retaining an incoming edge with the largest weight pointing to that node; checking whether there is a cycle in the selected edge set; if there is no cycle, the maximum spanning tree graph has been found; if there is a cycle, shrinking each cycle into a super node, and recalculating the weights of new edges entering and leaving the super node based on the weights of the edges inside and outside the cycle; recursively executing the aforementioned steps in the newly generated graph until there are no cycles, and finally expanding all super nodes to restore the tree structure.

[0018] Preferably, the conversion to a structured data stream in a preset format includes: starting from the virtual root node, performing a depth-first traversal of the globally optimal tree logical structure; during the traversal, generating a hierarchical data representation object for each document object unit; the hierarchical data representation object contains multiple core key-value pairs, namely, an identifier key as a unique identifier for the node, a type key for the node category label predicted by the classification head preset in the decoder, a bounding box key as the two-dimensional coordinate position information of the node, a text key for storing the corresponding text content when the node is a text block, and a list storing all child node data representation objects in traversal order, namely, child node keys; and outputting the hierarchical data representation object corresponding to the virtual root node of the entire document as a complete structured data stream.

[0019] Preferably, the element-wise addition of the original attention score matrix A and the bias matrix B includes: expanding the bias matrix B to the multi-head attention dimension using a tensor broadcasting mechanism to obtain an expanded bias matrix; and performing an element-wise addition operation between the scaled multi-head attention score matrix corresponding to the original attention score matrix A and the expanded bias matrix.

[0020] Secondly, the present invention provides a multimodal fusion complex layout document structure generation system, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned multimodal fusion complex layout document structure generation method is implemented.

[0021] By adopting the above technical solution, a computer program is generated from the above-mentioned method for generating complex layout documents using multimodal fusion, and stored in a memory for loading and execution by a processor. This allows for the creation of a terminal device based on the memory and processor, making it convenient to use.

[0022] The technical solution of the present invention has the following beneficial technical effects: This invention achieves a profound understanding of document content by deeply fusing multimodal information such as the two-dimensional coordinate position vector, visual region feature vector, layout attribute vector, and content semantic information vector of document object units. During the encoding stage, the model can construct specific spatial association ranges based on the layout and semantic features of each document object unit, thereby accurately detecting the essential differences in layout associations between different types of units and improving the ability to perceive the inherent logical connections within complex layouts. During the decoding stage, by introducing virtual root nodes and transforming the logical structure generation problem into a task of solving a globally optimal tree diagram, it ensures that a globally optimal, unique, and structurally complete tree logical structure can be determined from all potential parent-child relationships. This global optimization mechanism avoids problems such as structural inconsistencies or logical breaks caused by local decision-making errors, improving the accuracy of structurally generated complex layout documents. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a method for generating complex layout documents using multimodal fusion according to the present invention. Figure 2 This is a schematic diagram of a two-dimensional Gaussian space bias dynamically generated based on multimodal features in an embodiment of the present invention; Figure 3 This is a schematic diagram of the global optimization process for the logical hierarchy of complex layout documents in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0025] This invention discloses a method for generating complex layout documents using multimodal fusion, referring to... Figure 1 This includes steps S1-S4: S1: Parse the document to obtain a set of document object units; extract the two-dimensional coordinate position information vector and visual feature vector for each document object unit, and extract the content semantic information vector and layout attribute vector when the document object unit is a text block.

[0026] First, the target PDF or image document is opened using a document parsing library, and a page-by-page traversal is performed. During the traversal, vector graphics are extracted by calling the extract vector graphics method within the page object, images are extracted by calling the extract image method, and text is extracted by calling the extract text method and setting flag parameters to preserve the original font information, thereby extracting and obtaining the original text block document object units.

[0027] Vector graphics are obtained by calling the page object's drawing extraction method, the image extraction method is obtained by calling the image extraction method, and the text extraction method is called while configuring the flag parameter to preserve font information, in order to extract the original text block. For each document object unit, the x-coordinate of the upper left corner and the y-coordinate of the lower right corner are obtained as two-dimensional coordinate position information through the corresponding rectangle object in the document parsing library.

[0028] To extract visual region features, the page region corresponding to the coordinate position of the document object unit is rendered into a pixel map using the pixel map rendering method. This pixel map is then input into a pre-trained deep residual network model or visual transformer model. The feature map of the last layer or the pooling layer is taken and subjected to global average pooling to obtain the visual region feature vector.

[0029] For each text block unit, the plain text content contained within the unit is input into a pre-trained BERT model. The semantic information vector representing the content is obtained by performing average pooling on the word vector sequence output by the BERT model, or by using the hidden state of the initial classification label. Simultaneously, from the detailed text block information returned by the document parsing library, the font size and font flag attributes of each text segment are directly read to determine whether the font weight is bold. Line spacing is determined by calculating the difference in baseline coordinates of consecutive text lines. The values ​​corresponding to the font size, font weight, and line spacing are normalized and combined into a layout attribute vector.

[0030] In one embodiment, for each document object unit, two-dimensional coordinate position information and visual region features are extracted, and when the document object unit is a text block, content semantic information vector information and layout attributes are extracted, including: For each document object unit, extract the coordinates of the top-left and bottom-right vertices as a two-dimensional coordinate position information vector; The pixel region of each document object unit is input into a ResNet-50 convolutional neural network pre-trained on the ImageNet dataset, and the 2048-dimensional vector output by the last convolutional layer of the convolutional neural network is extracted as the visual region feature vector. For each text block unit, the BERT model is used to convert the text content into a 768-dimensional vector, which serves as the content semantic information vector. In addition, for text block units, the font size value, the corresponding mapping value of the font weight, and the ratio of line spacing to font size are extracted in points to form a layout attribute vector.

[0031] The ResNet-50 convolutional neural network is a deep residual network containing fifty layers. This deep residual network consists of an initial convolutional and pooling layer and four residual blocks. Each residual block contains multiple convolutional layers, batch normalization layers, linear rectified activation functions, and shortcut connections that enable cross-layer connections. The network input is a three-channel image tensor that has been size-normalized and pixel-value-normalized; its actual size can be set to 224×224×3. In the output feature stage, the global average pooling layer and fully connected classification layer from the original model are removed. The feature map output from the last convolutional block is used instead, and global average pooling is performed on it to obtain a 2048-dimensional feature vector.

[0032] The BERT model is a bidirectional encoder based on the Transformer architecture. Its network structure consists of multiple stacked Transformer encoder units, each integrating a multi-head self-attention module and a feedforward neural network module. The input data is a sequence of word indexes obtained after text segmentation, with a special start classification label added to the beginning of the sequence. The model generates a high-dimensional hidden state vector for each word in the input sequence, where the hidden state vector corresponding to the start classification label is used as the aggregate semantic representation of the entire text sequence, for example, a 768-dimensional vector. Those skilled in the art should know that if dimensional misalignment or mismatch occurs, dimensional alignment can be flexibly achieved using pooling, upsampling, or downsampling, which will not be elaborated further.

[0033] The document pages are traversed using a PDF parsing tool to identify and extract individual document object units. For each document object unit, its origin is recorded with the top-left corner of the page as the reference point. The bounding box coordinates in the coordinate system, specifically the coordinates of the top-left vertex. and the coordinates of the bottom right vertex This forms a four-dimensional vector of two-dimensional coordinate position information. Based on these bounding box coordinates, the pixel region corresponding to the document object unit is cropped from the rendered image of the document page, and the pixel region is uniformly scaled to 224×224 pixels. After standardization, this pixel region is input into a ResNet-50 model with the fully connected classification layer removed. Global average pooling is performed on the feature map output by the last convolutional block of the ResNet-50 model to finally obtain a 2048-dimensional visual region feature vector.

[0034] If the identified document object unit is a text block, its contained text string is input into a pre-trained BERT model. The 768-dimensional hidden state vector corresponding to the initial classification label in the model output is extracted as the semantic information vector of the text block. Simultaneously, layout attributes are extracted by parsing the metadata of the document object unit, specifically including the font size in points and the numerically mapped font weight information, for example, mapping regular font weight to 400 and bold font weight to 700. Furthermore, the ratio of line spacing to font size needs to be calculated; if the line spacing is 18 points and the font size is 12 points, the ratio is 1.5. Finally, the font size, font weight mapping value, and line spacing ratio are normalized to form the layout attribute vector of the text block.

[0035] Thus, this step, by parsing the document and extracting the two-dimensional coordinates, visual features, and semantic and layout attributes of specific units, achieves precise stripping of the underlying information of complex layouts, providing a high-quality, highly representative multimodal data foundation for the decoder to construct a document hierarchy transformation based on a globally optimal tree-like logical structure.

[0036] S2: Input the extracted features into a pre-trained model containing encoders and decoders; in the encoder, a feature space correlation weighted attention mechanism is used to generate a parameter set of spatial relation distribution based on the corresponding features for each source document object unit; calculate the spatial relation bias term based on the parameter set and inject it into the original attention score matrix for calculating self-attention.

[0037] In an optional embodiment, the main body of the model employs an encoder-decoder network based on the Transformer architecture. In the encoder part, all feature vectors of each document object unit are concatenated into a fused feature vector. When the multi-head self-attention module calculates the attention score, for each source document object unit used as a query term, the corresponding fused feature vector is fed into a three-layer multilayer perceptron. The output layer of this multilayer perceptron network is planned to have five neurons, each corresponding to one of the five parameters of a two-dimensional Gaussian distribution, i.e., the first mean. Second mean Raw output of the first standard deviation Raw output of the second standard deviation And the raw output of the correlation coefficient. For non-text block units such as images or vector graphics, their layout attributes and content semantic information vector information are filled with a zero vector or a learnable embedding vector of a specific type before being input into the multilayer perceptron network. For any target document object unit that serves as a key, the two-dimensional coordinates of the center of the target document object unit are substituted into the two-dimensional Gaussian probability density function defined by the five parameters generated by the source document object unit to calculate a probability density value, which is the spatial relation bias term. The spatial relation bias terms calculated between all source document object units and target document object units are used to form a bias matrix. This bias matrix is ​​then added element-wise to the dot product of the scaled attention score matrix (i.e., the query matrix and the transpose of the key matrix), and then a normalized exponential function operation is performed to complete the injection of the bias term.

[0038] In one embodiment, a parameter set of a two-dimensional Gaussian distribution is generated based on the two-dimensional coordinate position information vector of the source document object unit, the visual region feature vector, the layout attribute vector, and the content semantic information vector, including: The two-dimensional coordinate position information vector, visual region feature vector, layout attribute vector and content semantic information vector of the source document object unit are concatenated to obtain a fused feature vector. When the source document object unit is a non-text block, a preset placeholder vector is used to replace the layout attribute vector and content semantic information vector. The fused feature vector is input into a three-layer multilayer perceptron, which outputs five parameter values: the first mean of a two-dimensional Gaussian distribution. Second mean First standard deviation Second standard deviation and correlation coefficient ; Among them, the first standard deviation Second standard deviation Apply the Softplus activation function to the correlation coefficient By applying the hyperbolic tangent activation function, the above five parameters uniquely determine a two-dimensional Gaussian distribution.

[0039] A three-layer multilayer perceptron is a feedforward neural network. The network structure consists of an input layer, two hidden layers using linear rectified activation functions, and a linear output layer. The network input is a one-dimensional fused feature vector composed of the coordinates, visual features, layout, and semantic information of document object units. The network output is a vector containing five scalar values. The standard deviation is processed using a Softplus activation function, and the correlation coefficient is processed using a hyperbolic tangent activation function, corresponding to the five parameters of a two-dimensional Gaussian distribution, namely the first mean. Second mean First standard deviation Second standard deviation and correlation coefficient .

[0040] For a text block that serves as a source document object unit, a 2823-dimensional fusion feature vector is formed by concatenating a 4D coordinate vector, a 2048-dimensional visual region feature vector, a 3D layout attribute vector, and a 768-dimensional content semantic information vector. For non-text block units such as images or vector graphics, since these units lack layout attributes and content semantic information vectors, a pre-defined zero vector with the same dimensions as the layout and semantic vectors is used as a placeholder and concatenated with the unit's 4D coordinate vector, also forming a 2823-dimensional fusion feature vector. This fusion feature vector is then input into a three-layer multilayer perceptron. The input layer of this multilayer perceptron has a dimension of 2823, the first hidden layer has a dimension of 256 and uses a linear rectified activation function, the second hidden layer has a dimension of 128 and uses a linear rectified activation function, and the output layer has a dimension of 5 and does not use an activation function. The five neurons in the output layer correspond to the five initial parameters of a Gaussian distribution, namely the first mean. Second mean Raw output of the first standard deviation Raw output of the second standard deviation and the raw output of the correlation coefficient To ensure the reliability of the parameters, the raw output of the first standard deviation is... Raw output of the second standard deviation Apply the Softplus function to ensure the final generated first standard deviation Second standard deviation Always positive; raw output of the correlation coefficient. Applying the hyperbolic tangent function ensures the final correlation coefficient is generated. The value range is between [-1, 1]. For example, when the fused feature vector of a page title text block is input into a multilayer perceptron, a specific set of initial parameters may be generated, at which point the first mean... =0, second mean The value is 15, the first standard deviation. 300, second standard deviation The correlation coefficient is 10. The value of 0.1 defines a flat Gaussian distribution with a wide horizontal influence, indicating that the model logically tends to focus on other document object units that are on the same horizontal line and are vertically close to the page title text block.

[0041] Specifically, the first standard deviation satisfies the following relationship: ; in, Indicates the first standard deviation; Represent the natural logarithm function; Represents an exponential function with the natural constant as its base; This represents the raw output of the first standard deviation.

[0042] Specifically, the second standard deviation satisfies the following relationship: ; in, Indicates the second standard deviation; Represent the natural logarithm function; Represents an exponential function with the natural constant as its base; This represents the raw output of the second standard deviation.

[0043] Specifically, the correlation coefficient satisfies the following relationship: ; in, Represents the correlation coefficient; Represents the hyperbolic tangent function; This represents the raw output of the correlation coefficient.

[0044] In one embodiment, a spatial relation bias term is calculated based on a two-dimensional Gaussian distribution, and this spatial relation bias term is injected into the original attention score matrix used to calculate self-attention, including: For any source document object unit i and target document object unit j, calculate the center point coordinate offset of target document object unit j relative to source document object unit i. Substituting the center point coordinate offset into the two-dimensional Gaussian distribution generated by the source document object unit i, the value of the two-dimensional Gaussian distribution after removing the dimensionless exponent of the normalization coefficient is calculated and used as the spatial relation bias term. ; The spatial relationship bias terms between all document object unit pairs together form the bias matrix B; When calculating the self-attention score, the original attention score matrix A is added element-wise to the bias matrix B to obtain a new attention score matrix. Then apply the new attention score matrix Perform normalized exponential function processing.

[0045] For source document object unit i and target document object unit j, the center point coordinates of source document object unit i are obtained by calculating the average of the bounding box vertex coordinates of source document object unit i. The center point coordinates of the target document object unit j are obtained by calculating the average of the bounding box vertex coordinates of the target document object unit j. Next, the relative difference between the coordinates of the two center points on the horizontal and vertical axes is calculated to obtain the relative offset in the first dimension. Second dimension relative offset Then, the relative offset of the first dimension is... Second dimension relative offset Substituting the parameters of the two-dimensional Gaussian distribution generated from the source document object unit i into the quadratic form operation, a dimensionless intermediate variable z is calculated, and then combined with the correlation coefficient. Further conversion yields the spatial relation bias term. This spatial relation bias calculation logic is executed on all document object pairs, resulting in an N×N bias matrix B, where N is the total number of document object pairs. In the encoder's self-attention calculation, the original attention score matrix A is calculated by performing a dot product operation on the transposes of the query matrix Q and the key matrix K and dividing by a scaling factor. This original attention score matrix A is then element-wise added to the spatial relation bias matrix B to obtain a new attention score matrix that incorporates spatial relational information. Optionally, the bias matrix B can be extended to the multi-head attention dimension using a tensor broadcasting mechanism, and then element-wise added to the already scaled multi-head attention score matrix. Finally, the new attention score matrix... Applying a normalized exponential function to each row yields the final attention weight matrix.

[0046] Specifically, the coordinates of the center point satisfy the following relationship: ; ; in, This represents the coordinates of the center point of cell i in the source document object; This represents the coordinates of the center point of unit j in the target document object; and These represent the horizontal and vertical coordinates of the top-left vertex of the source document object unit i, respectively; and These represent the horizontal and vertical coordinates of the lower right corner vertex of the source document object unit i, respectively. and These represent the horizontal and vertical coordinates of the top-left vertex of the target document object unit j, respectively. and These represent the horizontal and vertical coordinates of the lower right corner vertex of the target document object unit j, respectively.

[0047] Specifically, the relative offset satisfies the following relationship: ; ; in, This represents the relative offset of the first dimension; This represents the relative offset of the second dimension; and These represent the horizontal coordinates of the center points of the target document object unit j and the source document object unit i, respectively. and These represent the vertical coordinates of the center points of the target document object unit j and the source document object unit i, respectively.

[0048] Specifically, the dimensionless intermediate variable satisfies the following relation: ; Where z represents a dimensionless intermediate variable; This represents the relative offset of the first dimension; This represents the first mean; Indicates the first standard deviation; Represents the correlation coefficient; This represents the relative offset of the second dimension; Indicates the second mean; This represents the second standard deviation.

[0049] Specifically, the spatial relation bias term satisfies the following relation: ; in, z represents the spatial relationship bias term of target document object unit j relative to source document object unit i; z represents a dimensionless intermediate variable. This represents the correlation coefficient.

[0050] Specifically, the original attention score matrix satisfies the following relation: ; Where A represents the original attention score matrix; Q represents the query matrix; and K represents the key matrix; The transpose of the key matrix; This represents the scaling factor dimension constant.

[0051] Specifically, the new attention score matrix satisfies the following relation: ; in, A represents the new attention score matrix; B represents the original attention score matrix; and C represents the bias matrix.

[0052] Thus, this step generates a dynamic two-dimensional Gaussian space bias term by fusing multimodal features with a feature space correlation weighted attention mechanism, overcoming the shortcomings of traditional attention in recognizing complex spatial dependencies, and laying a high-dimensional structural foundation for the decoder to generate a globally optimal logic tree.

[0053] S3: Input the embedding vector into the decoder as the virtual root node. For each document object unit, generate a candidate probability distribution with other units or the virtual root node as the parent node based on its decoding output state and the decoding output state of other units, including the virtual root node. Construct a weighted directed graph based on the parent node candidate probability distribution among all document object units. Use a graph search algorithm to determine the unique globally optimal tree logic structure with the virtual root node as the root in the weighted directed graph.

[0054] Initialize a learnable initial embedding vector with the same dimension as the decoder output state as the representation of the virtual root node, and add the initial embedding vector to the set of decoder output states of all document object units. For each document object unit whose parent node is to be determined, the decoder output state of the unit is used as the representation of the child node; the decoder output states of all other units in the set and the virtual root node are used as the representations of the candidate parent nodes. To calculate the probability of the parent-child relationship, a dual affine attention mechanism is used. First, two different linear transformation layers, i.e., two independent multilayer perceptrons, are used to process the child node representation and the candidate parent node representation respectively, resulting in two new vectors; then, a bilinear transformation is used to calculate the score between the two new vectors, which represents the probability that there is a parent-child relationship between the candidate parent node and the child node. Apply a normalized exponential function to the score vectors calculated for a child node and all candidate parent nodes to obtain a probability distribution, where each probability value corresponds to the probability that a candidate node is a parent node. After calculating the candidate probability distribution of parent nodes for all units, a weighted directed graph is constructed. The vertices of this graph represent all document object units and the virtual root node. The weight of the directed edge from the candidate parent node to the child node is the calculated parent-child relationship probability. The Julius Edmunds algorithm is applied, using the virtual root node as the starting point of the graph, to find a maximum spanning tree within this weighted directed graph. This tree represents the globally optimal tree structure. The Julius Edmunds algorithm is a greedy algorithm that finds the minimum or maximum spanning tree starting from a specified root in a weighted directed graph. This means that each non-root node has exactly one incoming edge and no cycles. The algorithm solves this by repeatedly selecting the minimum or maximum incoming edge for each node and shrinking directed cycles. Furthermore, the decoder output state of each document object unit is input into a classification header, which outputs the probability distribution of the node in a predefined category. The node with the highest probability is selected as its category label.

[0055] In one embodiment, generating a candidate probability distribution with each other unit and the virtual root node as the parent node includes: The hidden state vector of each document object unit output by the decoder, along with the learnable embedding vector representing the virtual root node, are input into a bilinear classifier. For any document object unit that is the target document object unit, the bilinear classifier receives its corresponding hidden state vector and calculates it sequentially with the learnable embedding vector corresponding to the virtual root node and the hidden state vectors corresponding to all other units, outputting a set of probability values ​​for the target document object unit. Each probability value in the set corresponds to the possibility of another unit or virtual root node being the parent node, and the set of probability values ​​together constitutes the parent node candidate probability distribution.

[0056] A bilinear classifier is a model used to calculate the score between two vectors. At its core is a learnable weight matrix. Suppose two vectors of the same dimension represent the first hidden state vectors of candidate parent nodes. and the second hidden state vector of the target child node The bilinear classifier obtains a scalar score for a candidate parent node as the parent node of the target child node by calculating a bilinear form. The scalar score This is then used to calculate the conditional probability of the parent-child relationship. Assume the decoder outputs a set of N hidden state vectors for N object units in the document. Each hidden state vector has a feature dimension of d, for example, d=768. Simultaneously, an initial embedding vector with the same feature dimension d is defined, which can be randomly initialized and trained along with the model. This serves as the representation of a virtual root node. It is used to predict arbitrary target document object units. The parent node of the target document object unit As a dependency, and its corresponding second hidden state vector As input, the set of hidden state vectors of all units, including the virtual root node, is used as the candidate representation of the parent node. The bilinear classifier contains a learnable weight matrix. The dimension is d×d. For each candidate parent node... Where i ranges from 0 to N, using its corresponding first hidden state vector Calculate the candidate parent node As a unit of the target document object Scalar score of the parent node For the target document object unit This yields a score vector containing N+1 scores. Applying a normalized exponential function to the score vector yields a probability distribution sequence, where each probability element in the sequence is... Indicates the corresponding candidate parent node It is the unit of the target document object. The probability of the parent node. This probability distribution sequence is the target node. The probability distribution of candidate parent nodes of a node.

[0057] Specifically, the scalar score satisfies the following relation: ; in, This represents the scalar score of the candidate parent node as the parent node of the target child node; This represents the first hidden state vector; This represents the transpose of the first hidden state vector; Represents the learnable weight matrix; This represents the second hidden state vector.

[0058] Specifically, the probability elements satisfy the following relation: ; in, Represents probability elements; This means that the scalar scores of all candidate nodes are summed after applying an exponential function. Represents an exponential function with the natural constant as its base; Indicates scalar score; This represents the scalar score of the k-th candidate parent node as the parent node of the target child node; This represents the scalar score of the candidate parent node as the parent node of the target child node; Indicates the candidate parent node; Indicates the target child node; Indicates the first One candidate parent node; Indicates the index of the candidate parent node; Indicates the index of the target child node; This represents the index used for summing and traversing candidate parent nodes.

[0059] In one embodiment, a weighted directed graph is constructed based on the probability distribution of candidate parent nodes among all document object units, including: Let the set of document object units be the first set U, and the virtual root node be... Then the set of vertices in the weighted directed graph is the first set U and the virtual root node. The union of; For any two vertices in a weighted directed graph, i.e., the source node and target node The source node Belongs to the set of vertices and target node For elements belonging to the first set U, and whose index i is not equal to j, construct a path from the source node. Point to target node The directed edges; Set the weight of the directed edge to the source node predicted by the decoder. The target node The conditional probability value of the parent node.

[0060] Suppose there are N object units in the document, which together constitute the first set mentioned above. Using virtual root nodes Construct a weighted directed graph G=(V,E,W). The set of vertices of the weighted directed graph objects. It contains N+1 vertices. For the target node in the graph... and any source node And since the coordinate index i is not equal to j, construct a path from the source node. Point to target node directed edges The directed edge edge weight Set as the source node calculated in the previous step As the target node The conditional probability value of the parent node. This forms an almost fully connected directed graph object, where, except for the virtual root node, for each non-root node, there are N incoming edges pointing to that node. These N incoming edges come from all other N nodes, including the virtual root node. There are only outgoing edges, no incoming edges.

[0061] In one embodiment, the maximum spanning tree graph algorithm is used to determine the unique globally optimal tree logical structure rooted at the virtual root node in a weighted directed graph, including: The Zhu Liu Edmunds algorithm is used to select and retain the incoming edge with the largest weight pointing to each node except the virtual root node. Check if there is a cycle in the selected edge set. If there is no cycle, then the maximum spanning tree has been found. If a cycle exists, shrink each cycle into a super node and recalculate the weights of new edges entering and leaving the super node based on the weights of the edges inside and outside the cycle. The aforementioned steps are recursively executed in the newly generated graph until there are no loops. Finally, all supernodes are expanded to restore the tree structure.

[0062] In the constructed weighted directed graph object, for all nodes except the virtual root node... Every non-root target node outside Select the input edge with the largest weight. And retain them. In the set of N retained edges, check if a cycle exists. This can be done by starting from any node and backtracking along the unique input edge; if you can return to that node itself, a cycle is found. If no cycles are detected, then the N edges and the virtual root node... Together they form a virtual root node The algorithm terminates when the root is the maximum spanning tree. If one or more cycles are detected, for each cycle... Perform a shrinking operation to merge all nodes in the loop into a single supernode. This forms a smaller, contracted graph. For a loop entering the original diagram The input edge from the external source node to the internal target node. The source node Target node Create a new incoming edge in the shrinking graph object. The weight of the first incoming edge of this edge is equal to the weight of the original input edge minus the weight of the loop. The difference between the weight of an internal edge pointing to the same internal target node and the minimum weight of the loop. For a line leaving the loop in the original graph... The output edge of the internal source node pointing to the external target node In the shrinking graph object, create a new outgoing edge, with the weight of its second outgoing edge being directly equal to the weight of the original outgoing edge. Recursively execute the algorithm on the shrinking graph object, starting from the steps above, until a maximum spanning tree without loops is found. When a new incoming edge is selected as the edge leading to a supernode, it indicates that a loop has been found in the original graph. In this process, edges pointing to the internal target node corresponding to the incoming edge and originating from within the loop are broken, while input edges originating from external source nodes are preserved. Following this rule, all shrunk nodes and edges are restored, ultimately yielding a globally optimal tree-like logical structure.

[0063] Specifically, the weight of the first incoming edge satisfies the following relation: ; in, Indicates the weight of the first incoming edge; Indicates the weights of the original input edges; Indicates a loop The weight of internal edges pointing to the same internal target node; Indicates a loop The minimum edge weight among all edge weights.

[0064] Specifically, the weights of the second outgoing edges satisfy the following relation: ; in, Indicates the weight of the second outgoing edge; This represents the original output edge weight.

[0065] Thus, this step generates candidate probabilities of parent-child relationships for global nodes through dual affine attention, and combines them with the maximum spanning tree graph algorithm to find the optimal tree, eliminating the error accumulation problem caused by autoregressive serial decoding and generating a unique and globally optimal tree logic structure.

[0066] S4: Based on the globally optimal tree-like logical structure, the document object unit and its corresponding logical hierarchy relationship are converted into a structured data stream in a preset format.

[0067] Starting from the virtual root node of the maximum spanning tree obtained in the previous step, a depth-first search algorithm is executed for traversal. During the traversal, a stack structure is maintained to record the node access paths, thereby representing the logical hierarchy. When a new node is accessed, a hierarchical data representation object is generated to represent that node according to a structured data format, such as hierarchical object representation. The original information of the node, such as its text content, data type, and two-dimensional coordinates, is filled into this hierarchical data representation object. Subsequently, all child nodes of the node in the tree structure are recursively accessed, and the set of child node data representation objects returned by the recursive call is used as a sub-element array of the current hierarchical data representation object. When all child nodes of a node have been accessed, the recursion is returned, completing the construction of the current hierarchical data representation object. The entire traversal process starts from the root node and continues until all nodes have been accessed once, generating a complete, nested hierarchical object representation data structure that can represent the logical hierarchy of the document; this is the final structured data flow.

[0068] In one embodiment, the document object units and logical hierarchy relationships are converted into a structured data stream of a preset format by performing a topological sorting traversal on the tree-like logical structure, including: Starting from the virtual root node, a depth-first traversal is performed on the tree-like logical structure. During the traversal, a data representation object is generated for each document object unit. The hierarchical data representation object contains several core key-value pairs: an identifier key as a unique identifier for the node, a type key for the node category label predicted by the classification head, a bounding box key for the node's two-dimensional coordinate position information, a text key for storing the corresponding text content when the node is a text block, and a list of all child node data representation objects in traversal order, i.e., the child node keys.

[0069] The hierarchical data representation objects corresponding to the root node of the entire document are output as a complete structured data stream.

[0070] The classification head is a feedforward neural network used for node type prediction. A feedforward neural network typically consists of a linear layer and an activation layer using a normalized exponential function. The input to the feedforward neural network is the hidden state vector output by the decoder from the document object unit, and the output is a probability distribution sequence for predefined node categories. The category with the highest probability in the probability distribution sequence is selected as the type label for that node. The classification head is jointly trained end-to-end with the encoder-decoder backbone network.

[0071] The above process is implemented through a recursive function, which starts from the virtual root node. Begin depth-first traversal. Continue traversing until any document object cell, i.e., the current node, is reached. Time: refers to the current node Create a hierarchical data representation object. This hierarchical data representation object specifically contains: an identifier key, the value of which is the current node. A unique numeric index; a type key whose value is used to access the current node. The hidden state vector in the decoder is predicted by the aforementioned classification head, and its preset category can be paragraph, title, list item, image, or table, etc.; the bounding box key is a list sequence containing 4 floating-point elements. Specifically, the horizontal coordinates of the top left corner vertex are as follows: Vertical coordinates of the top left corner vertex Horizontal coordinates of the bottom right vertex and the vertical coordinates of the bottom right corner vertex If the current node If the data is a text block, then the data representation object at this level also contains a text key, the value of which is the corresponding plain text content. Further, a child node key is added to this data representation object, and the value of this child node key is initialized to an empty list sequence. Then, the current node is traversed in a predetermined order. This function is called recursively on all child nodes in the tree-like logical structure. The child node data representation object returned by the recursive call to each child node is then appended sequentially to the current node. The hierarchical data represents the list of child nodes of the object. After traversal is complete, the constructed current node is returned. The complete hierarchical data representation object. The initial call to the entire topological sorting traversal process is to the aforementioned virtual root node. The execution output is a virtual root node. The corresponding hierarchical data representation object contains the entire list of child nodes. The list of child nodes includes the hierarchical data representation objects of all top-level logical units in the document, thus ultimately generating a complete structured data flow with a clear hierarchical structure.

[0072] Thus, this step generates a nested structured data stream by traversing the globally optimal tree-like logical structure through depth-first search, combined with a classification prediction head and a recursive call mechanism. This bridges the gap between visual typography and machine-readable logical structures, achieving a complete closed loop from visual typography to machine-readable logical structures, thereby improving the accuracy and completeness of document parsing.

[0073] To further summarize the overall system processing logic, a document parsing tool is used to identify document object units such as text blocks, images, and vector graphics within the document. For each document object unit, the system extracts the bounding box vertex coordinates of its page position. The pixel region corresponding to the document object unit is cropped and scaled to 224 x 224 pixels, and then input into a deep residual network model pre-trained on a public visual dataset to extract a 2048-dimensional visual feature vector. For text blocks, a bidirectional encoder model based on the Transformer architecture is used to encode the content into a 768-dimensional content semantic information vector, and the font size value, font weight (e.g., 400 for regular weight, 700 for bold weight), and the measured ratio of line spacing to font size are parsed to form a 3-dimensional layout attribute vector.

[0074] The core of the encoder is a feature space correlation-weighted attention mechanism. For each source document object unit, the concatenated multimodal feature vector is fed into a three-layer perceptron. This perceptron generates a two-dimensional Gaussian distribution with five parameters for that source document object unit, namely the first mean. Second mean First standard deviation Second standard deviation and correlation coefficient The standard deviation is guaranteed to be positive using a linear rectified activation function with a smooth approximation, and the correlation coefficient is constrained to the range [-1, 1] using a hyperbolic tangent function. For any target document object cell, the coordinate offset of its center point relative to the center point of the source document object cell is calculated. Then, this coordinate offset is substituted into the Gaussian distribution formula generated by the source document object cell to calculate a spatial relation offset term matrix. The spatial relation bias term matrix Add element-wise to the original attention score matrix Above, a new attention score matrix is ​​formed. Then, it is normalized using a normalized exponential function. This makes the attention weights not only dependent on content similarity, but also aware of spatial correlations of different ranges and directions determined by the layout and semantics of the source document object units.

[0075] At the decoder, the model aims to construct a globally optimal document logic tree. To this end, a learnable embedding vector is input as the virtual root node. The decoder outputs a hidden state vector for each document unit. The hidden state vector, along with the embedding vector of the virtual root node, is used to predict parent-child relationships. A bilinear classifier calculates the score indicating a parent-child relationship between any two nodes, and for each potential child node, the scores of all possible parent nodes are converted into a probability distribution using a normalized exponential function. The probability values ​​of this distribution are used as edge weights in a weighted directed graph, where the vertices are all document units and the virtual root node. The Zhu-Liu Edmunds algorithm is used to find a spanning tree rooted at the virtual root node with the maximum sum of all edge weights on this weighted directed graph. Through greedy selection, loop detection, shrinking, recursion, and expansion, a unique globally optimal tree logic structure is found. Starting from the virtual root node, a depth-first traversal is performed on the tree, transforming each node and its hierarchical relationship into a nested hierarchical object representation data structure. This structure includes the node's identifier index, the category label predicted by the independent classification head, the bounding box coordinates, the text content, and a list containing all child node data representation objects, thereby generating the final structured data stream.

[0076] During the experimental validation phase, the experimental conditions were set to train and evaluate on the PubLayNet dataset, which contains complex document page layouts. All models were implemented using an open-source deep learning framework on four GPUs, with a batch parameter size of 16, employing an adaptive momentum optimization algorithm, a learning rate of 0.0001, and a total of 50 training epochs. Evaluation metrics included tree edit distance (lower edit distance indicates more accurate structure prediction) and the average classification score for logical category prediction (higher average classification score indicates better classification performance).

[0077] The experiment included four models for comparative testing, specifically one complete scheme model and three ablation variant models: The first model was the complete scheme model, which integrated four features: coordinate position, visual features, semantic information, and layout attributes, and adopted a Gaussian distribution-based spatial relationship bias attention mechanism; the second model was the first ablation variant model, which ablated semantic information and layout attribute features, and the Gaussian bias term was generated only from coordinate position information; the third model was the second ablation variant model, which ablated the dynamic Gaussian bias mechanism and replaced it with a static two-dimensional relative position encoding strategy; and the fourth model was the third ablation variant model, which ablated visual features and used only coordinate position, semantic information, and layout attribute features.

[0078] Experimental results show that the first model, with its complete scheme, achieved a tree edit distance of 15.8 and a classification average score of 95.2%. In contrast, the second model, after absolving semantic and layout features, experienced a performance decrease, with the tree edit distance increasing to 23.1 and the classification average score dropping to 92.5%. The third model, after absolving the dynamic Gaussian bias mechanism, saw its tree edit distance increase to 21.5 and its classification average score reach 93.1%. The fourth model, after absolving visual features, achieved a tree edit distance of 18.2 and a classification average score of 94%.

[0079] Comprehensive analysis shows that the complete solution model performs best among all comparison items, verifying the overall performance superiority of the proposed method. Compared with the third model that ablates dynamic Gaussian bias, the complete solution reduces the tree edit distance by 5.7, demonstrating that generating spatial bias terms from multimodal features allows the attention mechanism to more flexibly detect spatial dependencies between different types of units, which is a core element in improving the accuracy of logical structure prediction. Compared with the second model that ablates semantic and layout features, the complete solution reduces the tree edit distance by 7.3, indicating that content semantic information vector information and layout attribute features are key data inputs for generating a logically consistent two-dimensional Gaussian distribution. Without these features, the model cannot accurately determine the logical hierarchy and correlation range between units. The fourth model that ablates visual features also leads to performance degradation, indicating that the visual low-level information extracted by the deep residual network provides indispensable low-level feature supplementary information for distinguishing non-textual feature elements such as images and tables, as well as detecting subtle visual layout differences between text blocks.

[0080] Figure 2 This is a schematic diagram of a two-dimensional Gaussian spatial bias dynamically generated based on multimodal features in an embodiment of the present invention. The diagram visually illustrates the core computational principle of the feature spatial correlation weighted attention mechanism. The contour lines in the diagram represent the two-dimensional Gaussian distribution generated by the source document object unit based on its features, with the center position of the source document object unit marked by a solid dot. Vertical and horizontal dashed lines indicate that the first mean of this distribution is 0 and the second mean is 15, respectively. Another solid dot in the diagram represents the target document object unit, and an arrow indicates the target's coordinate offset relative to the source, i.e., the first-dimensional relative offset and the second-dimensional relative offset. By substituting this coordinate offset into the aforementioned two-dimensional Gaussian distribution, the spatial relationship bias term between the target and the source, i.e., the correlation strength, can be calculated. This correlation strength adaptively reflects the spatial radiation influence range of the source node in the horizontal and vertical directions as the contour distribution changes.

[0081] Figure 3 This diagram illustrates the global optimization process for the logical hierarchy of complex document layouts in an embodiment of the present invention. It shows the decoding evolution from local probability prediction to the determination of the globally optimal structure. Figure 3 The left-hand stage A shows the constructed weighted directed graph, whose vertex set includes the virtual root node at the top and multiple document object unit nodes below, such as text block title nodes, text block paragraph nodes, and non-text block image nodes. The lines between the nodes represent potential parent-child pointing relationships, and the probability values ​​marked next to the lines represent the conditional probability values ​​predicted by the decoder. Figure 3 Phase B on the right side shows the result of solving the weighted directed graph using a maximum spanning tree algorithm, such as the Zhu Liu Edmunds algorithm. After processing, conflicting loops and redundant incoming edges are removed, ultimately forming a unique, globally optimal tree logical structure with a virtual root node as the starting point and no intersecting lines between nodes.

[0082] This invention also discloses a multimodal fusion complex layout document structure generation system, including a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement a multimodal fusion complex layout document structure generation method according to the present invention.

[0083] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0084] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for generating structured documents with complex layouts through multimodal fusion, characterized in that, include: Parse the document to obtain a collection of document object units; For each document object unit, extract two-dimensional coordinate position information vectors and visual feature vectors, and when the document object unit is a text block, extract content semantic information vectors and layout attribute vectors. Input the extracted features into a pre-trained model containing encoders and decoders. In the encoder, a feature space correlation weighted attention mechanism is used to generate a parameter set of spatial relationship distribution based on the corresponding features for each source document object unit. The spatial relationship bias term is calculated based on the parameter set and injected into the original attention score matrix for calculating self-attention. In the decoder, the embedding vector is input as a virtual root node. For each document object unit, based on its decoding output state and the decoding output states of other units including the virtual root node, a candidate probability distribution with other units or the virtual root node as parent nodes is generated. A weighted directed graph is constructed based on the parent node candidate probability distribution among all document object units. A graph search algorithm is used to determine the unique globally optimal tree logical structure with the virtual root node as the root in the weighted directed graph. Based on the globally optimal tree logical structure, the document object units and their corresponding logical hierarchy relationships are converted into a structured data stream in a preset format. The parameter set for generating the spatial relationship distribution includes: concatenating the two-dimensional coordinate position information vector, visual region feature vector, layout attribute vector, and content semantic information vector of the source document object unit to obtain a fused feature vector; when the source document object unit is a non-text block, a preset placeholder vector is used instead of the layout attribute vector and content semantic information vector; the fused feature vector is input into a three-layer multilayer perceptron, and the multilayer perceptron outputs five parameter values: the first mean of the two-dimensional Gaussian distribution, which is the specific form of the spatial relationship distribution. Second mean First standard deviation Second standard deviation and correlation coefficient ; where the first standard deviation Second standard deviation Apply the Softplus activation function to the correlation coefficient By applying the hyperbolic tangent activation function, the above five parameters uniquely determine a two-dimensional Gaussian distribution.

2. The method for generating complex layout documents using multimodal fusion according to claim 1, characterized in that, The step of extracting a two-dimensional coordinate position information vector and a visual feature vector for each document object unit, and extracting a content semantic information vector and a layout attribute vector when the document object unit is a text block, includes: for each document object unit, extracting the vertex coordinates of its bounding box as a two-dimensional coordinate position information vector; inputting the pixel region of each document object unit into a pre-trained visual feature extraction network to extract the visual feature vector; for text block units, using a pre-trained language model to convert the text content into the content semantic information vector; and for text block units, extracting the font size value, the mapping value corresponding to the character weight, and the ratio of line spacing to font size, which together constitute the layout attribute vector.

3. The method for generating complex layout documents using multimodal fusion according to claim 1, characterized in that, The step of calculating the spatial relationship bias term based on the parameter set and injecting it into the original attention score matrix for calculating self-attention includes: for any source document object unit i and target document object unit j, calculating the center point coordinate offset of target document object unit j relative to source document object unit i; substituting the center point coordinate offset into the two-dimensional Gaussian distribution generated by source document object unit i, and calculating the value of the two-dimensional Gaussian distribution after removing the dimensionless exponent of the normalization coefficient, as the spatial relationship bias term. The spatial relationship bias terms between all document object pairs are used to form the bias matrix B. When calculating the self-attention score, the original attention score matrix A and the bias matrix B are added element-wise to obtain the new attention score matrix. Then apply the new attention score matrix Each row is processed using a normalized exponential function.

4. The method for generating complex layout documents using multimodal fusion according to claim 1, characterized in that, The generation of candidate probability distributions with other units or virtual root nodes as parent nodes includes: inputting the hidden state vector of each document object unit output by the decoder, and the learnable embedding vector representing the virtual root node, into a bilinear classifier; for any document object unit that is a target document object unit, the bilinear classifier receives its corresponding hidden state vector, and sequentially calculates it with the learnable embedding vector corresponding to the virtual root node and the hidden state vectors corresponding to all other units, to output a set of probability values ​​for the target document object unit; wherein each probability value in the set of probability values ​​corresponds to the possibility of one of the other units or the virtual root node being a parent node, and the set of probability values ​​together constitute the candidate probability distribution of the parent node.

5. The method for generating complex layout documents using multimodal fusion according to claim 1, characterized in that, The step of constructing a weighted directed graph based on the candidate probability distribution of parent nodes among all document object units includes: setting the set of document object units as the first set U, and the virtual root node as... Then the set of vertices in the weighted directed graph is the first set U and the virtual root node. The union of the two vertices in a weighted directed graph; for any two vertices in the graph, i.e., the source node. and target node The source node Belongs to the set of vertices and target node For elements belonging to the first set U, and whose index i is not equal to j, construct a path from the source node. Point to target node The directed edges; the weight of the directed edges is set to the weight of the source node predicted by the decoder. The target node The conditional probability value of the parent node.

6. The method for generating complex layout documents using multimodal fusion according to claim 1, characterized in that, The method of using a graph search algorithm to determine the unique globally optimal tree logical structure rooted at the virtual root node in a weighted directed graph includes: using the Julius Edmonds algorithm, for each node except the virtual root node, selecting and retaining an incoming edge with the largest weight pointing to that node; checking whether there is a cycle in the selected edge set; if there is no cycle, the maximum spanning tree graph has been found; if there is a cycle, shrinking each cycle into a super node, and recalculating the weights of new edges entering and leaving the super node based on the weights of the edges inside and outside the cycle; recursively executing the aforementioned steps in the newly generated graph until there are no cycles, and finally expanding all super nodes to restore the tree structure.

7. The method for generating complex layout documents using multimodal fusion according to claim 1, characterized in that, The conversion to a structured data stream in a preset format includes: starting from the virtual root node, performing a depth-first traversal of the globally optimal tree logical structure; during the traversal, generating a hierarchical data representation object for each document object unit; the hierarchical data representation object contains multiple core key-value pairs, namely, an identifier key as a unique identifier for the node, a type key for the node category label predicted by the classification head preset in the decoder, a bounding box key as the two-dimensional coordinate position information of the node, a text key for storing the corresponding text content when the node is a text block, and a list storing all child node data representation objects in traversal order, namely, child node keys; and outputting the hierarchical data representation object corresponding to the virtual root node of the entire document as a complete structured data stream.

8. The method for generating complex layout documents using multimodal fusion according to claim 3, characterized in that, The element-wise addition of the original attention score matrix A and the bias matrix B includes: expanding the bias matrix B to the multi-head attention dimension using a tensor broadcasting mechanism to obtain the expanded bias matrix; and adding the scaled multi-head attention score matrix corresponding to the original attention score matrix A to the expanded bias matrix element-wise.

9. A multimodal fusion system for generating complex layout documents, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement a method for generating complex layout documents using multimodal fusion as described in any one of claims 1-8.

Images