A fine-grained entity extraction method and system for complex documents

Abstract

The application belongs to the technical field of natural language processing and information extraction, and specifically relates to a fine-grained entity extraction method and system for complex documents, which comprises the following steps: obtaining an OCR text box set of a document image, extracting text content and coordinate information of each text box, and fusing the text features and visual features to generate a multi-modal fusion feature matrix; calculating geometric position relationships according to the coordinate information of each text box, modeling and fusing spatial correlations by using a multi-scale kernel function to generate a spatial bias matrix; dividing all attention heads into geometric constraint heads and global free heads, calculating attention and fusing after multi-head linear projection to obtain context-enhanced features; and calculating label emission scores, combining sequence constraint values to optimize model parameters, and outputting entity extraction results through global optimal sequence search. The application does not require external pre-training data, has small parameter quantity and fast reasoning speed, and can be used for entity extraction of complex documents such as government forms and commercial bills.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and information extraction technology, specifically a fine-grained entity extraction method and system for complex documents. Background Technology

[0002] Rich visual documents, such as paper documents, electronic forms, official documents, and business contracts, are widely used. To digitize and automate office processes, it is necessary to automatically extract key entity information such as names, ID numbers, mailing addresses, amounts, dates, and organization names from documents with complex layouts. Fine-grained entity extraction technology has thus become a core component of intelligent document processing. These documents generally exhibit characteristics such as dense field arrangement, inconsistent layout styles, scattered key-value pairs, and multi-level table nesting. Furthermore, scanned and photographed documents are prone to image distortion and light and shadow interference. Combined with text deviations and detection box position offsets generated during optical character recognition (OCR), this further increases the difficulty of accurate entity recognition. Currently, the industry still commonly uses manual data entry or fixed template matching. This approach not only requires significant manpower and has low processing efficiency but also cannot adapt to documents with flexible layouts, has a low error tolerance rate, and struggles to meet the demands of large-scale, automated document processing.

[0003] To address the shortcomings of manual processing methods, those skilled in the art have successively developed document entity extraction schemes based on deep learning, mainly including four categories: sequence labeling models, graph structure modeling models, multimodal pre-trained models, and lightweight document processing models. Among them, sequence labeling models only perform semantic analysis on one-dimensional text sequences, without utilizing the two-dimensional layout and spatial location information of documents, and are prone to problems such as incorrect entity boundary determination and confusion of adjacent field content under complex layouts; graph structure modeling models rely on manually constructed rules to construct the topological relationship of text units, which can only represent discrete connection features and cannot describe the continuous geometric relationship between text boxes, and have poor resistance to recognition errors and image distortion, and the computational cost is also relatively high; although multimodal pre-trained models have strong feature representation capabilities, they rely on massive pre-training data and high-end computing power support, and the model size is huge, making them unsuitable for application scenarios involving confidential data and limited computing power; existing lightweight models weaken spatial structure modeling capabilities to reduce deployment difficulty, and cannot take into account both local field alignment and global semantic relationship, resulting in significant shortcomings in extraction accuracy. In summary, existing technical solutions cannot simultaneously achieve high-precision extraction, lightweight deployment, strong anti-interference capabilities, and low resource adaptability, becoming a technical bottleneck restricting the intelligent implementation of complex documents. Therefore, it is urgent to propose a new entity extraction method to overcome the shortcomings of existing technologies. Summary of the Invention

[0004] The purpose of this invention is to provide a fine-grained entity extraction method and system for complex documents. It acquires text and visual fusion features from OCR units through lightweight multimodal feature embedding, models the continuous spatial correlation of text box geometric positions using multi-scale kernel functions to generate a spatial bias matrix, obtains context-enhanced features by dividing the attention head into a geometrically constrained head and a globally free head through hybrid multi-head attention interaction, and finally optimizes label sequence consistency through CRF global decoding to output the entity extraction results. This invention requires no external pre-training data, has a small parameter count, and fast inference speed, making it suitable for fine-grained entity extraction from complex documents such as government forms, commercial invoices, or official contracts.

[0005] The technical solution adopted by this invention to achieve the above objectives is: a fine-grained entity extraction method for complex documents, comprising the following steps:

[0006] Step S1: Obtain the set of OCR text boxes of the document image, extract the text content and coordinate information of each text box, obtain the text features and visual features of each OCR unit, fuse the text features and visual features, generate the multimodal fusion features of each OCR unit, and form a multimodal fusion feature matrix.

[0007] Step S2: Calculate the geometric positional relationship based on the coordinate information of each text box, use a multi-scale kernel function to model the spatial correlation of the geometric positional relationship and adaptively fuse it to generate spatial constraint information, and then convert it into a spatial bias matrix;

[0008] Step S3: Divide all attention heads into a first group and a second group, perform multi-head linear projection on the multimodal fusion feature matrix, calculate attention based on the spatial bias matrix for the first group of attention heads, calculate attention for the second group of attention heads, fuse the outputs of all attention heads to obtain context-enhanced features;

[0009] Step S4: Calculate the label emission score of each OCR unit based on context-enhanced features. During the training phase, combine the real class sequence and sequence constraint values ​​to calculate the sequence score and training loss to optimize the model parameters. During the inference phase, solve for the optimal label sequence through global optimal sequence search and output the entity extraction result.

[0010] Step S1 specifically includes:

[0011] Step S1-1: Obtain the OCR text box collection: ;

[0012] Wherein, a single text box is defined as , This indicates the OCR text box number and its value. , This represents the total number of text boxes. Indicates the first The text content of each text box. , They represent the first The coordinate quadruple of each text box is, in order, the x-coordinate of the top left corner, the y-coordinate of the top left corner, the width, and the height;

[0013] Step S1-2: Concatenate all text boxes in numerical order. Obtain the text sequence The text feature matrix is ​​obtained by encoding with a shallow Transformer encoder, i.e.:

[0014]

[0015] in, For encoder layers, For feature dimension, Indicates only before use The shallow BERT encoder of the layer, This represents the output text semantic feature matrix, where each row corresponds to the text feature of one OCR unit;

[0016] Steps S1-3: Map document images to visual features using Patch Embedding Where M is the total number of image patch blocks, according to The corresponding image patch is cropped according to coordinates to obtain the first... Local visual features of a text box ;

[0017] Steps S1-4: Perform linear transformation and layer normalization on the text features and local visual features to fuse them, obtaining the first... Multimodal fusion features of OCR units ,Right now:

[0018]

[0019] in, The weight matrix is ​​a linear transformation of text features. The weight matrix is ​​a linear transformation of visual features. To fuse the bias vector, Characteristic matrix The Middle row vectors For the first Local visual features of a text box Presentation layer normalization operation;

[0020] Steps S1-5: Final output of the fused feature matrix .

[0021] Step S2 specifically includes:

[0022] Step S2-1: Calculate the first... The geometric center coordinates of the text boxes are:

[0023]

[0024]

[0025] in, They represent the first The coordinates of the top left corner, the top left corner, the width of the text box, and the height of the text box are in a quadruple set.

[0026] Step S2-2: Based on the total width of the document page Total height For any two text boxes Calculate the normalized squared central distance ,Right now:

[0027]

[0028] Step S2-3: Settings Using Gaussian kernels with different bandwidths, a single-scale spatial correlation is constructed, namely:

[0029]

[0030] in, Indicates the Gaussian scale index, with values ​​ranging from 1 to 10. , For the first Gaussian kernel bandwidth parameter, This represents the natural exponential function. Indicates the first Text box under scale Spatial correlation values;

[0031] Step S2-4: Introduce learnable weights to obtain normalized fusion weights. :

[0032]

[0033] in, These are learnable parameters;

[0034] Step S2-5: According to Obtain overall spatial correlation;

[0035] Step S2-6: Convert spatial correlation into a geometric bias matrix and obtain the elements of the geometric bias matrix. ,Right now:

[0036]

[0037] in, Represents a minimal constant with a fixed value. , Represent the natural logarithm function;

[0038] Step S2-7: Embed the geometric bias matrix into the attention score calculation to obtain the attention score with fused geometric bias. ,Right now:

[0039]

[0040] in, Indicates the first The query vector corresponding to each text box feature. Indicates the first The key vector corresponding to the features of each text box. For single-attention head feature dimensions;

[0041] Step S2-8: Obtain the final attention weights after Softmax normalization. for:

[0042]

[0043] in, Indicated by The Softmax normalization function is for the dimension .

[0044] Step S3 includes the following steps:

[0045] Step S3-1: Gather all attention points Divide into two disjoint sets: the geometric constraint head set and the geometric constraint head set. and global free head set ;

[0046] Step S3-2: Perform multi-head linear projection on the multimodal fusion feature matrix Z to obtain the first... The query, key, and value feature matrix of the head , , They are respectively:

[0047]

[0048]

[0049]

[0050] Where h is the number of attention heads. , , , Both are projection weight matrices;

[0051] Step S3-3: Calculate the geometric constraint head set respectively. and global free head set Attention weights and outputs; weight the outputs of all attention heads according to... Fuse into a context-enhanced feature matrix , This is for outputting the projection matrix.

[0052] Step S3-3 includes the following steps:

[0053] a) For the geometric constraint head, superimpose the geometric bias matrix, and calculate the attention weights and output, respectively:

[0054]

[0055]

[0056] in, Indicates the single-head feature dimension. For learnable scaling factor, The geometric bias matrix, For the geometrically constrained head, normalize the attention weight matrix. Output the feature matrix for the geometric constraint head;

[0057] b) For the global free head, without introducing geometric bias, the attention weights and output are calculated using standard self-attention, as follows:

[0058]

[0059]

[0060] in, The attention weight matrix is ​​a global free head normalized matrix. Output the feature matrix for the global free head.

[0061] Step S4 specifically includes:

[0062] Step S4-1: Map the contextual features to label emission scores through a linear classification layer to obtain the first... Tag transmission score per OCR unit for:

[0063]

[0064] in, for The OK, Represents the classification layer weight matrix; This represents the bias vector for the classification layer. , Indicates the total number of entity tag categories;

[0065] Step S4-2: Calculate the total score for the label path, i.e.:

[0066]

[0067] in, To input all features of the document, To predict the label sequence, For the first Entity tags assigned to each OCR unit, For the first Unit label The corresponding launch fraction, For tags Switch to tab The score;

[0068] Step S4-3: During the training phase, construct the CRF training loss function;

[0069] Step S4-4: In the inference phase, the Viterbi algorithm is used to solve for the globally optimal label sequence, i.e.:

[0070]

[0071] in, This represents the globally optimal entity label sequence, i.e., the final fine-grained entity extraction result. This indicates that the Viterbi algorithm solves the path score. The largest label sequence, Indicates input The set of all corresponding valid label sequences.

[0072] 7. A fine-grained entity extraction method for complex documents according to claim 6, characterized in that, in step S4-3, the construction of the CRF training loss function specifically comprises:

[0073]

[0074] in, This represents the overall training loss of the model. This represents the true labeled sequence of the sample. Represents any valid label sequence. Represents the natural index. Represents the natural logarithm. s Let be the scores for the calculated real path and the arbitrary path, respectively.

[0075] An extraction system for a fine-grained entity extraction method for complex documents includes:

[0076] The multimodal feature embedding module is used to obtain the set of OCR text boxes of the document image, extract the text content and coordinate information of each text box, obtain the text features and visual features of each OCR unit, fuse the text features and visual features, generate the multimodal fusion features of each OCR unit, and form a multimodal fusion feature matrix.

[0077] The spatial offset calculation module is used to calculate the geometric positional relationship based on the coordinate information of each text box, and to perform spatial correlation modeling and adaptive fusion of the geometric positional relationship using a multi-scale kernel function to generate spatial constraint information, which is then converted into a spatial offset matrix.

[0078] The hybrid attention interaction module is used to divide all attention heads into a set of geometrically constrained heads and a set of global free heads, perform multi-head linear projection on the multimodal fusion feature matrix, calculate attention based on the spatial bias matrix of the geometrically constrained heads, calculate attention based on the global free heads, and fuse the outputs of all attention heads to obtain context-enhanced features;

[0079] The sequence decoding module is used to calculate the label emission score of each OCR unit based on context-enhanced features. During the training phase, it combines the real class sequence and sequence constraint values ​​to calculate the sequence score and training loss to optimize the model parameters. During the inference phase, it solves for the optimal label sequence through global optimal sequence search and outputs the entity extraction result.

[0080] The present invention has the following beneficial effects and advantages:

[0081] 1. This invention employs multi-scale Gaussian geometric bias to model the two-dimensional spatial relationship of OCR text boxes in a continuous manner, effectively resisting interference such as OCR offset and image distortion, and reducing entity misclassification and confusion problems.

[0082] 2. This invention combines a hybrid multi-head attention mechanism, which can simultaneously take into account local field alignment and global long-distance semantic interaction, resulting in stronger feature representation capabilities. Tests show that this invention achieves F1 scores of 0.804, 0.766, and 0.854 on the XFUND-ZH, FUNSD, and Gov-CN datasets, respectively, demonstrating significantly better recognition accuracy than traditional sequence models and graph structure models.

[0083] 3. This invention features a lightweight architecture with only 62M parameters, resulting in fast inference speed, low hardware overhead, and no reliance on large-scale external pre-training data. It is suitable for environments with strict data control, edge devices, and low resource requirements. The addition of a CRF module to constrain the label sequence further enhances the coherence and reasonableness of the recognition results. The model demonstrates stable performance through cross-validation, exhibiting outstanding robustness and generalization ability, and can be widely applied to complex document entity extraction scenarios such as government affairs, finance, and contracts. Attached Figure Description

[0084] Figure 1 A schematic diagram of the overall process of the fine-grained entity extraction method for complex documents of this invention;

[0085] Figure 2 A schematic diagram of Gaussian spatial correlation distribution in the multi-scale Gaussian geometric bias calculation of this invention;

[0086] Figure 3 Schematic diagram of the multi-scale Gaussian fusion mechanism of this invention;

[0087] Figure 4 A schematic diagram of the hybrid attention head response mechanism of this invention. Detailed Implementation

[0088] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. The following examples are for illustrative purposes only and should not be considered as limiting the scope of the invention.

[0089] This embodiment discloses a fine-grained entity extraction method for complex documents, which consists of four core steps: lightweight multimodal feature embedding, multi-scale Gaussian geometric bias calculation, hybrid multi-head attention feature interaction, and CRF tag global decoding. This method uses random initialization training, requiring no large-scale external pre-training data, and can be deployed in local operating environments with limited computing power and strict data control.

[0090] In a preferred embodiment of the present invention, the hyperparameters are uniformly fixed as follows:

[0091] (1) The optimizer used is AdamW, and the learning rate is set to Weight decay is set to First-order moment estimation of exponential decay rate Second-order moment estimation of exponential decay rate ;

[0092] (2) The batch size is set to 16 and the maximum training epoch is set to 100. An early stopping strategy is adopted, and training is stopped when the F1 value on the validation set does not improve for 10 consecutive epochs.

[0093] (3) Unified feature dimensions of the model The feedforward network has a dimension of 3072, and the encoder has [number of layers]. ;

[0094] (4) Total number of heads of multi-head attention There are 6 geometric constraint heads and 6 global free heads, in a 1:1 ratio;

[0095] (5) Number of Gaussian kernels at multiple scales Initial bandwidth parameters , , The model parameters are updated adaptively during training.

[0096] (6) Numerical stability constant It is used to prevent numerical overflow in logarithmic operations during geometric bias matrix calculations.

[0097] The unified and complete execution flow of the method of the present invention is as follows:

[0098] (I) Dataset Construction and Preprocessing

[0099] When implementing this invention, a labeled dataset needs to be constructed according to the specific application scenario. The dataset should be divided into a training set, a validation set, and a test set in a 7:1.5:1.5 ratio. Entity label categories are predefined according to business needs. For example, in government form scenarios, they can be defined as Header, Question, Answer, and Other; in commercial form scenarios, they can be defined as Key, Value, Header, and Other. The annotation method adopts the BIO or BIOES sequence annotation system, assigning a unique entity label to each OCR text box.

[0100] Document image preprocessing includes: uniformly scaling the image resolution to an appropriate size (e.g., maintaining aspect ratio, with the shorter side no less than 800 pixels), rotating and correcting tilted documents, and performing adaptive histogram equalization on images with uneven lighting. OCR recognition results need to filter out text boxes with excessively low confidence (e.g., confidence below 0.5) and merge overlapping text boxes using non-maximum suppression.

[0101] (II) Model Training Process

[0102] Step 1: Lightweight Multimodal Feature Embedding

[0103] Input a document image and read the image width. and height Call the OCR detection and recognition module to batch output all text boxes. Corresponding text content and coordinate quadruple Count the total number of text boxes .

[0104] Concatenate all text boxes in order of their numbers. Obtain the text sequence The 6-layer shallow BERT encoder is fed in and the formula is executed. The text feature matrix is ​​obtained. This shallow BERT encoder uses randomly initialized weights and does not load any externally pre-trained weights.

[0105] Patch Embedding is used to uniformly segment the document image into fixed-size image patches to obtain global visual features. ,in Total number of patches. Based on the coordinates of each text box. The corresponding image patch is cropped from the global visual features and obtained after linear mapping. Local visual features of a text box .

[0106] Execute multimodal fusion formula line by line Generate fusion features for each text box. The multimodal fusion feature matrix is ​​obtained by splicing the features together. Then, pass it to step two.

[0107] Step 2: Multi-scale Gaussian geometric bias calculation

[0108] Iterate through all text boxes Execute the formula , Calculate the geometric center coordinates of each text box. .

[0109] For any two text boxes Combined with the total width of the document images and total height Execute the formula Calculate the normalized squared central distance .

[0110] Pick Group Gaussian kernel bandwidth parameters , , Execute the formula Generate three single-scale spatial correlation matrices , , .

[0111] Execute formula Calculate the normalized fusion weights for each scale ,in These are learnable parameters, initialized to 0, and adaptively updated during training.

[0112] Execute formula Obtain the comprehensive spatial correlation matrix .

[0113] Introducing a minimum constant Execute the formula Generate geometric bias matrix .

[0114] The multimodal fusion features obtained in step one Linear projection of the query vector and key vector (The projection matrix is ​​shared with step three), execute the formula. Obtain the attention score with fused geometric bias, then execute the formula. Obtain normalized attention weights to complete the fusion of spatial constraints and attention weights.

[0115] Step 3: Hybrid Multi-Head Attention Feature Interaction

[0116] Focus all attention Divided into geometric constraint head set and global free head set , There are two groups of six heads each.

[0117] The fused feature matrix output from step one Substitute into the formula , , This yields the query, key, and value feature matrices for all 12 headers. , , .

[0118] For geometric constraint head ( Substitute the geometric bias matrix With learnable scaling factor Execute the formula , Output geometric constraint head features .

[0119] For global free head ( Without introducing geometric bias, the standard self-attention formula is applied. , Output global free head features .

[0120] Concatenate the outputs of all 12 attention heads and execute the formula. Obtain the context-enhanced feature matrix Then, pass it to step four.

[0121] Step 4: Global Decoding of CRF Tags

[0122] Context-enhanced feature matrix Each row vector Substitute into the formula The label emission score of each text box is obtained. , This represents the total number of entity tag categories.

[0123] Input manually labeled real label sequences Execute the formula Calculate the path score of the true label sequence, where The label transition matrix (learnable parameters, randomly initialized). Indicates the virtual label at the start of the sequence. A virtual label indicating the end of the sequence.

[0124] Execute the CRF training loss function Calculate training loss All model parameters (including shallow BERT encoder parameters and multimodal fusion parameters) are updated through backpropagation. , , Gaussian fusion weights Attention projection matrix , , , Geometric constraint head scaling factor Classification layer parameters , Label transition matrix wait).

[0125] During the inference phase, for unlabeled input documents, the Viterbi algorithm is used to execute the formula. Find the globally optimal label sequence that maximizes the path score. Output the final fine-grained entity extraction results.

[0126] This invention provides a fine-grained entity extraction system for complex documents, based on a fine-grained entity extraction method for complex documents, comprising:

[0127] The multimodal feature embedding module is used to obtain the set of OCR text boxes of the document image, extract the text content and coordinate information of each text box, obtain the text features and visual features of each OCR unit, fuse the text features and visual features, generate the multimodal fusion features of each OCR unit, and form a multimodal fusion feature matrix.

[0128] The spatial offset calculation module is used to calculate the geometric positional relationship based on the coordinate information of each text box, and to perform spatial correlation modeling and adaptive fusion of the geometric positional relationship using a multi-scale kernel function to generate spatial constraint information, which is then converted into a spatial offset matrix.

[0129] The hybrid attention interaction module is used to divide all attention heads into a set of geometrically constrained heads and a set of global free heads, perform multi-head linear projection on the multimodal fusion feature matrix, calculate attention based on the spatial bias matrix of the geometrically constrained heads, calculate attention based on the global free heads, and fuse the outputs of all attention heads to obtain context-enhanced features;

[0130] The sequence decoding module is used to calculate the label emission score of each OCR unit based on context-enhanced features. During the training phase, it combines the real class sequence and sequence constraint values ​​to calculate the sequence score and training loss to optimize the model parameters. During the inference phase, it solves for the optimal label sequence through global optimal sequence search and outputs the entity extraction result.

[0131] To verify the effectiveness and universality of this invention, three typical application scenarios were selected for implementation and verification: complex government documents, general Chinese forms, and English noisy forms with scanning distortion. These three scenarios correspond to the self-built anonymized government dataset Gov-CN, the public dataset XFUND-ZH, and FUNSD, respectively, covering various complex layout types such as Chinese government approval forms, Chinese and English business forms, and noisy documents with scanning distortion. All implementation cases adopted the aforementioned unified hyperparameters and complete four-step execution process. The models were all trained using random initialization without loading any external pre-trained weights.

[0132] Example 1: Entity Extraction from Complex Government Documents

[0133] This implementation case study applies to various government application forms and approval registration forms. It utilizes a self-built, anonymized government dataset, Gov-CN, divided into training, validation, and test sets in a 7:1.5:1.5 ratio. Entity labels are categorized into four types: Header, Question, Answer, and Other. The model training and inference are completed using the aforementioned four-step process. Test results show: precision 0.855, recall 0.854, and F1 score 0.854. The model has 62M parameters, a single-page inference time of 35.7ms, and a processing speed of up to 28 pages per second. It can be stably deployed on a regular intranet server, meeting the requirements for automated batch entry of government documents.

[0134] Example 2: General Chinese Form Entity Extraction

[0135] This implementation case focuses on general business forms and expense reimbursement documents, using the publicly available dataset XFUND-ZH. The samples exhibit diverse formats and uneven field distribution, leading to interference from conventional OCR recognition offsets. Entity labels are defined as Key, Value, Header, and Other. Experiments were conducted using unified parameters and a complete execution flow. Test metrics included: precision 0.810, recall 0.812, and F1 score 0.804. Compared to traditional graph structure models such as SDMG-R and MWGR, this invention demonstrates stronger adaptability to varied formats and significantly reduces entity boundary recognition errors.

[0136] Example 3: Extraction of English Noise Forms with Scanning Distortion

[0137] This implementation case uses the publicly available dataset FUNSD, which contains documents with interference such as image distortion, blurriness, and OCR recognition noise, to verify the model's robustness against interference. The model was run according to a unified four-step process, and the final F1 score was 0.766. Experiments demonstrate that this invention, relying on continuous Gaussian space constraints and hybrid multi-head collaborative attention, can maintain stable extraction results even under conditions of strong noise and irregular layouts.

[0138] As can be seen from the three sets of implementation cases, the unified execution process of this invention is simple and lightweight. It relies on multi-scale Gaussian geometric bias to solve the defects of discrete space modeling, and uses hybrid multi-head attention to achieve both local layout and global semantics. It does not require external pre-training data, and balances extraction accuracy, inference speed and deployment flexibility. It can be adapted to various complex document processing scenarios such as government affairs, finance, and foreign affairs documents, and has outstanding engineering application value.

[0139] Compared to existing technologies, this invention possesses several significant advantages. It employs multi-scale Gaussian geometric bias to continuously model the two-dimensional spatial relationships of OCR text boxes, effectively resisting interference from OCR offset and image distortion, and reducing entity misclassification and confusion. Combined with a hybrid multi-head attention mechanism, it can simultaneously consider local field alignment and global long-distance semantic interaction, resulting in stronger feature representation capabilities. Testing shows that this invention achieves F1 scores of 0.804, 0.766, and 0.854 on the XFUND-ZH, FUNSD, and Gov-CN datasets, respectively, demonstrating significantly superior recognition accuracy compared to traditional sequence models and graph structure models.

[0140] In summary, this invention achieves superior extraction accuracy compared to traditional graph structure models and sequence labeling models with a lightweight parameter count of only 62M, without requiring external pre-training data. It maintains stable performance across various scenarios, including government and commercial forms, and noisy documents. This invention employs multi-scale Gaussian geometric bias to continuously model the two-dimensional spatial relationships of OCR text boxes, effectively resisting interference from OCR offset and image distortion, and reducing entity misclassification and confusion. Combined with a hybrid multi-head attention mechanism, it simultaneously considers local field alignment and global long-distance semantic interaction, resulting in stronger feature representation capabilities. The addition of a CRF module to constrain the label sequence further enhances the coherence and reasonableness of the recognition results. The model demonstrates stable performance through cross-validation, exhibiting outstanding robustness and generalization ability. It can be widely applied to complex document entity extraction scenarios such as government affairs, finance, and contracts, demonstrating significant engineering application value.

[0141] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fine-grained entity extraction method for complex documents, characterized in that, Includes the following steps: Step S1: Obtain the set of OCR text boxes of the document image, extract the text content and coordinate information of each text box, obtain the text features and visual features of each OCR unit, fuse the text features and visual features, generate the multimodal fusion features of each OCR unit, and form a multimodal fusion feature matrix. Step S2: Calculate the geometric positional relationship based on the coordinate information of each text box, use a multi-scale kernel function to model the spatial correlation of the geometric positional relationship and adaptively fuse it to generate spatial constraint information, and then convert it into a spatial bias matrix; Step S3: Divide all attention heads into a first group and a second group, perform multi-head linear projection on the multimodal fusion feature matrix, calculate attention based on the spatial bias matrix for the first group of attention heads, calculate attention for the second group of attention heads, fuse the outputs of all attention heads to obtain context-enhanced features; Step S4: Calculate the label emission score of each OCR unit based on context-enhanced features. During the training phase, combine the real class sequence and sequence constraint values ​​to calculate the sequence score and training loss to optimize the model parameters. During the inference phase, solve for the optimal label sequence through global optimal sequence search and output the entity extraction result.

2. The fine-grained entity extraction method for complex documents according to claim 1, characterized in that, Step S1 specifically includes: Step S1-1: Obtain the OCR text box collection: ; Wherein, a single text box is defined as , This indicates the OCR text box number and its value. , This represents the total number of text boxes. Indicates the first The text content of each text box. , They represent the first The coordinate quadruples of each text box are, in order, the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the width, and the height; Step S1-2: Concatenate all text boxes in numerical order. Obtain the text sequence The text feature matrix is ​​obtained by encoding with a shallow Transformer encoder, i.e.: ； in, For encoder layers, For feature dimension, Indicates only before use The shallow BERT encoder of the layer, This represents the output text semantic feature matrix, where each row corresponds to the text feature of one OCR unit; Steps S1-3: Map document images to visual features using Patch Embedding Where M is the total number of image patch blocks, according to The corresponding image patch is cropped using coordinates to obtain the first... Local visual features of a text box ; Steps S1-4: Perform linear transformation and layer normalization on the text features and local visual features to fuse them, obtaining the first... Multimodal fusion features of OCR units ,Right now: ； in, The weight matrix is ​​a linear transformation of text features. The weight matrix is ​​a linear transformation of visual features. To fuse the bias vector, Characteristic matrix The Middle row vectors For the first Local visual features of a text box Presentation layer normalization operation; Steps S1-5: Final output of the fused feature matrix .

3. The fine-grained entity extraction method for complex documents according to claim 1, characterized in that, Step S2 specifically includes: Step S2-1: Calculate the first... The geometric center coordinates of the text boxes are: ； ； in, They represent the first The coordinates of the top left corner, the top left corner, the width of the text box, and the height of the text box are in a quadruple set. Step S2-2: Based on the total width of the document page Total height For any two text boxes Calculate the normalized squared central distance ,Right now: ； Step S2-3: Settings Using Gaussian kernels with different bandwidths, a single-scale spatial correlation is constructed, namely: ； in, Indicates the Gaussian scale index, with values ​​ranging from 1 to 10. , For the first Gaussian kernel bandwidth parameter, This represents the natural exponential function. Indicates the first Text box under scale Spatial correlation values; Step S2-4: Introduce learnable weights to obtain normalized fusion weights. : ； in, These are learnable parameters; Step S2-5: According to Obtain overall spatial correlation; Step S2-6: Convert spatial correlation into a geometric bias matrix and obtain the elements of the geometric bias matrix. ,Right now: ； in, Represents a minimal constant with a fixed value. , Represents the natural logarithm function; Step S2-7: Embed the geometric bias matrix into the attention score calculation to obtain the attention score with fused geometric bias. ,Right now: ； in, Indicates the first The query vector corresponding to each text box feature. Indicates the first The key vector corresponding to the features of each text box. For single-attention head feature dimensions; Step S2-8: Obtain the final attention weights after Softmax normalization. for: ； in, Indicates The Softmax normalization function for the dimension is .

4. The fine-grained entity extraction method for complex documents according to claim 1, characterized in that, Step S3 includes the following steps: Step S3-1: Gather all attention points Divide into two disjoint sets: the geometric constraint head set and the geometric constraint head set. and global free head set ; Step S3-2: Perform multi-head linear projection on the multimodal fusion feature matrix Z to obtain the first... The query, key, and value feature matrix of the head , , They are respectively: ； ； ； Where h is the number of attention heads. , , , Both are projection weight matrices; Step S3-3: Calculate the geometric constraint head set respectively. and global free head set Attention weights and outputs; weight the outputs of all attention heads according to... Fuse into a context-enhanced feature matrix , This is for outputting the projection matrix.

5. A fine-grained entity extraction method for complex documents according to claim 1, characterized in that, Step S3-3 includes the following steps: a) For the geometric constraint head, superimpose the geometric bias matrix, and calculate the attention weights and output, respectively: ； ； in, Indicates the single-head feature dimension. For learnable scaling factor, The geometric bias matrix, For the geometrically constrained head, normalize the attention weight matrix. Output the feature matrix for the geometric constraint head; b) For the global free head, without introducing geometric bias, the attention weights and output are calculated using standard self-attention, as follows: ； ； in, The attention weight matrix is ​​a global free head normalized matrix. Output the feature matrix for the global free head.

6. The fine-grained entity extraction method for complex documents according to claim 1, characterized in that, Step S4 specifically includes: Step S4-1: Map the contextual features to label emission scores through a linear classification layer to obtain the first... Tag transmission score per OCR unit for: ； in, for The OK, Represents the classification layer weight matrix; This represents the classification layer bias vector. , This indicates the total number of entity tag categories. Step S4-2: Calculate the total score for the label path, i.e.: ； in, To input all features of the document, To predict the label sequence, For the first Entity tags assigned to each OCR unit, For the first Unit label The corresponding launch fraction, For tags Switch to tab The score; Step S4-3: During the training phase, construct the CRF training loss function; Step S4-4: In the inference phase, the Viterbi algorithm is used to solve for the globally optimal label sequence, i.e.: ； in, This represents the globally optimal entity label sequence, i.e., the final fine-grained entity extraction result. This indicates that the Viterbi algorithm solves the path score. The largest label sequence, Indicates input The set of all corresponding valid label sequences.

7. A fine-grained entity extraction method for complex documents according to claim 6, characterized in that, In step S4-3, the construction of the CRF training loss function specifically involves: ； in, This represents the overall training loss of the model. This represents the true labeled sequence of the sample. Represents any valid label sequence. Represents the natural index. Represents the natural logarithm. s Let be the scores for the calculated real path and the arbitrary path, respectively.

8. The extraction system for a fine-grained entity extraction method for complex documents according to any one of claims 1-7, characterized in that, include: The multimodal feature embedding module is used to obtain the set of OCR text boxes of the document image, extract the text content and coordinate information of each text box, obtain the text features and visual features of each OCR unit, fuse the text features and visual features, generate the multimodal fusion features of each OCR unit, and form a multimodal fusion feature matrix. The spatial offset calculation module is used to calculate the geometric positional relationship based on the coordinate information of each text box, and to perform spatial correlation modeling and adaptive fusion of the geometric positional relationship using a multi-scale kernel function to generate spatial constraint information, which is then converted into a spatial offset matrix. The hybrid attention interaction module is used to divide all attention heads into a set of geometrically constrained heads and a set of global free heads, perform multi-head linear projection on the multimodal fusion feature matrix, calculate attention based on the spatial bias matrix of the geometrically constrained heads, calculate attention based on the global free heads, and fuse the outputs of all attention heads to obtain context-enhanced features; The sequence decoding module is used to calculate the label emission score of each OCR unit based on context-enhanced features. During the training phase, it combines the real class sequence and sequence constraint values ​​to calculate the sequence score and training loss to optimize the model parameters. During the inference phase, it solves for the optimal label sequence through global optimal sequence search and outputs the entity extraction result.

Images