Cross-modal document information extraction method based on space-semantic alignment

Abstract

The invention relates to a cross-modal document information extraction method based on space-semantic alignment, and belongs to the field of artificial intelligence, computer vision and natural language processing. According to the method, the spatial feature and semantic information bidirectional alignment model is designed, by constructing the spatial feature and semantic feature bidirectional alignment model, the document layout information can dynamically adjust attention distribution of text semantic features, meanwhile, semantic information reversely optimizes the spatial features, collaborative modeling of spatial layout and semantic information is achieved, and the document layout efficiency is improved. Therefore, the accuracy and robustness of complex document information extraction are improved. According to the method, a hierarchical cross-modal information extraction model is designed, through the hierarchical cross-modal information extraction model, the overall structure of a document is recognized on the global level, local key content is focused on the regional level, fine modeling is conducted on fine-grained texts and visual elements on the entity level, and accurate recognition of a cross-modal entity and the semantic relation of the cross-modal entity is achieved; and the generalization ability and applicability of information extraction are enhanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, computer vision and natural language processing, and particularly relates to a cross-modal document information extraction method based on space-semantic alignment. BACKGROUND

[0002] With the development of artificial intelligence and natural language processing technology, document understanding and information extraction are widely used in military, finance and other fields. Cross-modal document information extraction has important significance by jointly analyzing text, image and layout information to achieve efficient acquisition of structured information. Traditional methods rely on single modal features and only use text for entity or relationship extraction, ignoring the visual and layout information of the document, resulting in insufficient accuracy and robustness of information extraction for complex or structured documents.

[0003] In vertical fields and low-resource scenarios, the types of documents are complex and the labeled data is scarce. Traditional supervised learning methods lack generalization ability when dealing with new event types or complex layout, and it is difficult to meet the actual application requirements. The spatial layout information in the document is closely related to the text semantics. Spatial information not only carries the structure level and position characteristics of the document, but also has an important influence on the understanding and extraction results of the text semantics. However, the existing technology is difficult to fully capture the interaction between spatial layout and text semantics, and there are single modal limitations and cross-modal semantic deviation problems, which limit the accuracy and reliability of information extraction in low-resource environments.

[0004] The application proposes a cross-modal document information extraction method based on space-semantic alignment, which mainly solves three problems.

[0005] First, the single modal feature limitation problem. Existing technologies rely on text content for entity or relationship recognition, ignoring the visual and spatial layout information in the document, resulting in insufficient accuracy and robustness of information extraction in complex or structured documents. The application proposes to realize the collaborative modeling of cross-modal information through the deep fusion of spatial features and semantic information, and improve the understanding ability of complex document structure and semantics.

[0006] Second, the problem of insufficient interaction between spatial features and semantic information. Traditional methods are difficult to fully capture the interaction between spatial layout and text semantics in the document, and lack a dynamic adjustment mechanism, so that spatial information cannot effectively enhance semantic understanding, and semantic information is also difficult to optimize spatial features. The application introduces a bidirectional interaction attention mechanism, so that spatial features and semantic features can be adjusted and enhanced, realizing the bidirectional alignment of spatial layout and text semantics, and improving the precision of cross-modal information extraction.

[0007] Third, the multi-level information modeling is limited, and the traditional method usually lacks hierarchical modeling of the global structure, local area and fine-grained entity of the document, and it is difficult to capture key information at the global and local levels. The present application proposes a hierarchical cross-modal information extraction model, which identifies the document structure pattern at the global level, focuses on the local key information at the regional level, and models the fine-grained text and location elements at the entity level, to realize accurate identification of cross-modal entities and their semantic relationships, and improve the generalization ability and application value of information extraction.

[0008] The present application proposes a cross-modal document information extraction method capable of realizing spatial layout and semantic information collaborative modeling, which deeply fuses text representation and spatial features to improve the precision and generalization ability of complex document information extraction, and provides high-quality structured information support for military command decision-making, plan execution and other vertical fields. SUMMARY

[0009] (I) Technical problems to be solved

[0010] The technical problem to be solved by the present application is how to provide a cross-modal document information extraction method based on spatial-semantic alignment to solve the problems of single-modal feature limitation, insufficient interaction between spatial features and semantic information, and insufficient interaction between spatial features and semantic information.

[0011] (II) Technical scheme

[0012] In order to solve the above technical problems, the present application proposes a cross-modal document information extraction method based on spatial-semantic alignment, which comprises a spatial feature and semantic information bidirectional alignment stage and a hierarchical cross-modal information extraction stage.

[0013] In the spatial feature and semantic information bidirectional alignment stage, the document image is analyzed, the boundary box coordinates, relative position, layout format and adjacency relationship of the text area are extracted, and the spatial feature vector is converted; at the same time, the extracted text content is input into the pre-trained language model to obtain the semantic feature vector, and the attention mechanism is used to enhance the weight of the key semantic unit; on this basis, the bidirectional interaction attention mechanism is introduced, so that the spatial feature can dynamically adjust the attention distribution of the semantic feature, and the semantic information can also affect the weight of the spatial feature in the opposite direction, thereby realizing the bidirectional alignment between the spatial layout and the semantic information.

[0014] In the hierarchical cross-modal information extraction stage, the aligned spatial features and semantic features are fused at the global level to identify the overall structure pattern of the document, the local area is further focused on the basis of the identification result of the global structure, the cross-modal interaction mechanism is used to extract the regional level aggregation features, and the fine-grained text and visual elements in the region are further modeled at the entity level to realize accurate identification of cross-modal entities and their semantic relationships.

[0015] (Three) beneficial effects

[0016] The application proposes a cross-modal document information extraction method based on space-semantic alignment, and the main advantages are as follows:

[0017] (1) A bidirectional alignment model of spatial features and semantic information is designed, which can dynamically adjust the attention distribution of text semantic features by constructing a bidirectional alignment model of spatial features and semantic features, and the semantic information can also optimize the spatial features in the reverse direction, realizing the collaborative modeling of spatial layout and semantic information, and improving the accuracy and robustness of complex document information extraction.

[0018] (2) A hierarchical cross-modal information extraction model is designed, which can identify the overall structure of the document at the global level, focus on the local key content at the regional level, and finely model the fine-grained text and visual elements at the entity level, realize the accurate identification of cross-modal entities and their semantic relationships, and enhance the generalization ability and applicability of information extraction. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The flowchart of the application. DETAILED DESCRIPTION

[0020] In order to make the purpose, content and advantages of the application clearer, the specific embodiments of the application will be further described in detail below in combination with the drawings and examples.

[0021] The application belongs to the field of artificial intelligence, computer vision and natural language processing, and specifically relates to a cross-modal document information extraction technology based on a space feature and semantic information alignment mechanism.

[0022] In order to realize the deep fusion of text and position representation in the cross-modal document understanding task, the application proposes a cross-modal document information extraction method based on space-semantic alignment, which consists of two parts: 1) a bidirectional alignment model of spatial features and semantic information 2) a hierarchical cross-modal information extraction model. The method includes: a bidirectional alignment stage of spatial features and semantic information and a hierarchical cross-modal information extraction stage.

[0023] In the spatial feature and semantic information bidirectional alignment stage, the document image is layout parsed, the bounding box coordinates, relative position, layout format and adjacency relationship of the text region are extracted, and they are converted into spatial feature vectors; at the same time, the extracted text content is input into a pre-trained language model to obtain semantic feature vectors, and the attention mechanism is used to enhance the weight of the key semantic unit; on this basis, the bidirectional interaction attention mechanism is introduced, so that the spatial feature can dynamically adjust the attention distribution of the semantic feature, and the semantic information can also reversely affect the weight of the spatial feature, thereby realizing the bidirectional alignment between the spatial layout and the semantic information.

[0024] In the hierarchical cross-modal information extraction stage, the aligned spatial features and semantic features are fused at the global level to identify the overall structure mode of the document, the local area is further focused on according to the identification result of the global structure, the region-level aggregated features are extracted through the cross-modal interaction mechanism, and the fine-grained text and visual elements in the region are further modeled at the entity level to realize accurate identification of cross-modal entities and their semantic relationships. Through the above hierarchical information extraction method, the accuracy and generalization ability of cross-modal document information extraction are improved.

[0025] Step one: spatial-semantic alignment model construction

[0026] The spatial-semantic alignment model aims to realize the deep fusion of spatial layout information and text semantic information in the document image, thereby improving the accuracy and robustness of cross-modal document information extraction. The model mainly consists of two functional modules: (1) spatial feature encoding module; (2) semantic feature encoding and alignment module.

[0027] Firstly, the input document image is layout parsed to accurately extract the bounding box coordinates , relative position (coordinate proportion to the whole document , where is the width of the document image, is the height of the document image), layout format (font size and font style ) and adjacency relationship (represented by adjacency matrix , if text block is adjacent to text block , then , otherwise ). These spatial related information is encoded and mapped into a sequence of spatial feature vectors , where , d is the dimension, represents the spatial feature vector of the th text block, and m is the number of text blocks, which can be calculated as:

[0028] (1)

[0029] In the formula, represents a spatial feature encoder for encoding various spatial information into a vector of a unified dimension, A i is the adjacency vector corresponding to text block i (extracted from the adjacency matrix A, the first row).

[0030] At the same time, the text content in the document is input into the pre-trained language model Bert, and the sentence-level or paragraph-level text is mapped into a continuous semantic space to obtain a sequence of semantic feature vectors , wherein , corresponds to the semantic feature vector of the text block , as follows:

[0031] (2)

[0032] In the formula, is the text content of the i-th text block.

[0033] In order to realize the dynamic interaction between spatial features and semantic features, a bidirectional interaction attention mechanism is introduced. Through two layers of interaction attention structure, on the one hand, the spatial feature realizes the guidance of the attention distribution of the semantic feature, and on the other hand, the semantic feature realizes the reverse adjustment of the spatial weight. The attention weight of the spatial feature to the semantic feature The calculation process is as follows:

[0034] (3)

[0035] wherein, is the query representation of the semantic feature, obtained by linear transformation of the semantic feature , that is, ; is the key representation of the spatial feature, obtained by linear transformation of the spatial feature , that is, , and d is the feature dimension. represents the attention degree of the spatial feature to different positions of the semantic feature; similarly, get .

[0036] Through the above interaction mechanism, the spatial feature and the semantic feature are fused to obtain a sequence of fused cross-modal feature vectors , wherein The calculation is:

[0037] (4)

[0038] ​wherein is the vector of the is vector of the vector of the denotes element-wise multiplication. Through this interaction mechanism, the model can adaptively adjust the matching relationship between space and semantics, realizing the bidirectional alignment of layout and semantic information.

[0039] Step two: hierarchical cross-modal information extraction model

[0040] The goal of the hierarchical cross-modal information extraction model is to model the global structure, regional structure and local entity information of the document at each level based on aligned features, so as to realize multi-level information extraction from the whole document to fine-grained entity relationships. The model includes three levels, namely the global structure recognition layer, the regional feature focusing layer and the entity and relationship extraction layer.

[0041] First, the multi-head self-attention network is used to perform global feature fusion on the cross-modal feature sequence to obtain the global semantic expression of the document :

[0042] (5)

[0043] wherein MultiHeadSelfAttn represents multi-head attention, which is calculated by multiple attention heads in parallel to capture the long-distance dependency relationship between different positions in the feature sequence, thereby describing the organizational structure of the document from a macro perspective and providing context constraints for the positioning of local regions.

[0044] On the basis of the global level, the module introduces a regional attention module. Based on the global semantic expression , the module dynamically selects the attention region and performs weighted aggregation on the local key block, thereby realizing information focusing from global to regional. Its weight calculation can be expressed as:

[0045] (6)

[0046] wherein denotes the attention weight of the i-th region, is a learnable weight matrix, denotes the concatenation of the global feature G (d-dimensional) and the i-th cross-modal feature f i (d-dimensional).

[0047] Then, the calculation of the regional-level aggregated feature is as follows: ​

[0048] (7)

[0049] wherein, The important information of each local cross-modal feature under the guidance of the global structure is integrated.

[0050] At the entity level, based on the region-level aggregation feature R and the cross-modal feature vector sequence F, the entity class and the relationship type are predicted by a sequence labeling method (such as conditional random field CRF). For entity recognition, the entity label set is defined, and the entity label corresponding to each text block is predicted, and the probability calculation can be realized by a linear layer and a Softmax function:

[0051] (8)

[0052] wherein, and are learnable parameters. For relationship extraction, the relationship label set is defined, and the relationship label between the entity pair is predicted, .

[0053] In order to ensure the alignment consistency of the spatial-semantic features and the robustness of the cross-level extraction, the present application adopts a multi-task joint optimization strategy. The loss function is composed of two parts:

[0054] (9)

[0055] wherein, is a hyperparameter (the value range is 0≤λ≤1), and its core role is to balance the weight of the alignment loss and the extraction loss in the total loss. When is closer to 1, the model training will pay more attention to the alignment consistency of the spatial features and the semantic features (the constraint of is stronger), and the matching of the two modal features is ensured to be more accurate; when is closer to 0, the model training will pay more attention to the accuracy of the entity and relationship extraction (that is, the constraint of is stronger), and the core performance of the task is preferentially ensured.

[0056] For the alignment loss, which is used to constrain the matching consistency of the spatial features and the semantic features, the loss can be calculated by using the cosine similarity as follows:

[0057] (10)

[0058] For the extraction loss, including the entity loss and the relationship prediction loss. The entity loss can be calculated by using the cross-entropy as follows:

[0059] (11)

[0060] wherein, is the true entity label of the i-th text block.

[0061] The relation loss also adopts cross-entropy loss:

[0062] (12)

[0063] wherein, n is the number of entity pairs, E is the set of entity pairs, and ε is the set of all entity pairs to be predicted, represents traversing each pair of entities in the set , calculating the relation prediction loss of each pair of entities, and finally measuring the overall error of relation extraction through the cross-entropy loss function. is the true relation label of the entity pair (i, j), and the final loss is calculated as follows:

[0064] (13)

[0065] Through the joint training mechanism, the model can establish a stable alignment mapping between the spatial and semantic modalities, and realize semantic transmission and feature sharing in different levels of information extraction. Among them, The consistency of the model in matching spatial features and semantic features is strengthened, ensuring that the two can be accurately aligned; and Let the model accurately identify entities and their semantic relationships from cross-modal documents. The combination of these two losses enables the model to accurately and robustly complete the document information extraction task in the cross-modal scenario.

[0066] The spatial-semantic alignment-based cross-modal document information extraction method effectively solves the problem of insufficient fusion of spatial and semantic information in cross-modal documents by first realizing bidirectional alignment of spatial and semantic information and then performing hierarchical information extraction, better utilizing the multi-modal information of the document. At the same time, the spatial features and semantic features are interactively encoded, allowing the model to fully exploit the associated information of the spatial layout of the document image and the text semantics, obtaining more comprehensive knowledge from limited document data, thereby performing information extraction on unseen cross-modal documents, and having good generalization ability in cross-modal document understanding scenarios.

[0067] The application discloses a cross-modal document information extraction method based on spatial-semantic alignment, and the main advantages are as follows:

[0068] (1) A bidirectional alignment model of spatial features and semantic information is designed, by constructing a bidirectional alignment model of spatial features and semantic features, the layout information of the document can dynamically adjust the attention distribution of the text semantic features, at the same time the semantic information also reversely optimizes the spatial features, the collaborative modeling of spatial layout and semantic information is realized, so as to improve the accuracy and robustness of complex document information extraction.

[0069] (2) A hierarchical cross-modal information extraction model is designed, by the hierarchical cross-modal information extraction model, the global structure of the document is recognized at the global level, the local key content is focused at the regional level, and the fine-grained text and visual elements are finely modeled at the entity level, the accurate identification of cross-modal entities and their semantic relationships is realized, and the generalization ability and applicability of information extraction are enhanced.

[0070] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled persons in the technical field, several improvements and modifications can be made without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A method for cross-modal document information extraction based on spatial-semantic alignment, characterized in that, This method includes: a two-way alignment stage of spatial features and semantic information and a hierarchical cross-modal information extraction stage; In the bidirectional alignment stage of spatial features and semantic information, the document image is parsed to extract the bounding box coordinates, relative positions, layout formats, and adjacency relationships of text regions, and these are transformed into spatial feature vectors. Simultaneously, the extracted text content is input into a pre-trained language model to obtain semantic feature vectors, and an attention mechanism is used to enhance the weights of key semantic units. Based on this, a bidirectional interactive attention mechanism is introduced, enabling spatial features to dynamically adjust the attention distribution of semantic features, while semantic information can also influence the weights of spatial features, thereby achieving bidirectional alignment between spatial layout and semantic information. In the hierarchical cross-modal information extraction stage, aligned spatial and semantic features are fused at the global level to identify the overall structural pattern of the document. At the regional level, the recognition results of the global structure are further focused on local regions, and regional aggregated features are extracted through cross-modal interaction mechanisms. At the entity level, fine-grained text and visual elements within the region are further modeled to achieve accurate identification of cross-modal entities and their semantic relationships.

2. The cross-modal document information extraction method based on spatial-semantic alignment as described in claim 1, characterized in that, The step of parsing the document image layout, extracting the bounding box coordinates, relative positions, formatting, and adjacency relationships of text regions, and converting them into spatial feature vectors includes: First, the layout of the input document image is parsed to accurately extract the bounding box coordinates of each text region. Relative position, layout format (font size) and font styles And adjacency relationships; where relative position refers to the coordinate ratio of the entire document, including: ,in For document image width, Document image height; formatting includes: font size. and font styles Adjacency relationships are expressed using an adjacency matrix. This indicates that if the text block With text blocks Adjacent, then ,otherwise ; Encode this spatial information and map it into a sequence of spatial feature vectors. ,in d is the dimension. Indicates the first The spatial feature vectors of text blocks, where m is the number of text blocks, are calculated as follows: （1） In the formula, This represents a spatial feature encoder, used to encode multiple spatial information sources into a vector of a uniform dimension. A i It is the adjacency vector corresponding to text block i.

3. The cross-modal document information extraction method based on spatial-semantic alignment as described in claim 2, characterized in that, The process of inputting the extracted text content into a pre-trained language model to obtain semantic feature vectors includes: The text content of the document is input into the pre-trained language model BERT, which maps sentence-level or paragraph-level text into a continuous semantic space, resulting in a sequence of semantic feature vectors. ,in , Corresponding text block The semantic feature vector is as follows: （2） In the formula, It is the first The text content of each text block.

4. The cross-modal document information extraction method based on spatial-semantic alignment as described in claim 3, characterized in that, The introduction of a bidirectional interactive attention mechanism enables spatial features to dynamically adjust the attention distribution of semantic features, while semantic information can also inversely influence the weights of spatial features, thereby achieving bidirectional alignment between spatial layout and semantic information. To achieve dynamic interaction between spatial and semantic features, a bidirectional interactive attention mechanism is introduced. This mechanism employs a two-layer interactive attention structure: firstly, it guides the attention distribution of semantic features based on spatial features; secondly, it enables the reverse adjustment of spatial weights by semantic features. The attention weights of spatial features on semantic features are... The calculation process is as follows: （3） in, It is a query representation based on semantic features, derived from semantic features through a linear transformation. To obtain, that is ; It is a key representation of spatial features, derived from spatial features through a linear transformation. To obtain, that is d is the feature dimension; This indicates the degree of attention that spatial features pay to different positions of semantic features; similarly, it yields... ; Through the aforementioned interaction mechanism, spatial and semantic features are fused to obtain a fused cross-modal feature vector sequence. ,in The calculation is as follows: （4） In the formula, yes The Middle The vector of rows, yes The Middle The vector of rows, This indicates element-wise multiplication; through this interaction mechanism, the model can adaptively adjust the matching relationship between space and semantics, achieving bidirectional alignment of layout and semantic information.

5. The cross-modal document information extraction method based on spatial-semantic alignment as described in claim 4, characterized in that, The process of fusing aligned spatial and semantic features at the global level to identify the overall structural pattern of a document includes: First, a multi-head self-attention network is used to process cross-modal feature sequences. Perform global feature fusion to obtain the global semantic representation of the document. : （5） MultiHeadSelfAttn represents multi-head attention, which uses multiple attention heads to compute in parallel, capturing long-distance dependencies between different positions in the feature sequence, thereby macroscopically characterizing the document's organizational structure and providing contextual constraints for subsequent local region localization.

6. The cross-modal document information extraction method based on spatial-semantic alignment as described in claim 5, characterized in that, The step of further focusing on local regions based on the recognition results of the global structure at the region level, and extracting region-level aggregated features through a cross-modal interaction mechanism includes: Building upon the global level, a regional attention module is introduced; this module enhances global semantic expression. Based on this, the region of interest is dynamically selected, and local key blocks are weighted and aggregated to achieve information focusing "from global to regional"; the weight calculation is expressed as follows: （6） In the formula, Indicates the first Attention weights for each region It is a learnable weight matrix. This indicates that the global feature G and the i-th cross-modal feature f are combined. i To splice; Then, regional aggregation features The calculation is as follows: （7） in, It integrates important information from various local cross-modal features guided by the global structure.

7. The cross-modal document information extraction method based on spatial-semantic alignment as described in claim 6, characterized in that, The further modeling of fine-grained text and visual elements within the region at the entity level to achieve accurate identification of cross-modal entities and their semantic relationships includes: At the entity level, based on regional aggregated features R and cross-modal feature vector sequences F, entity categories and relation types are predicted through sequence labeling. For entity recognition, an entity label set is defined. Predict the entity label corresponding to each text block. Its probability calculation can be achieved through a linear layer and a softmax function: （8） In the formula, and These are learnable parameters; for relation extraction, a set of relation labels is defined. Predict the relationship labels between entity pairs ∈ .

8. The cross-modal document information extraction method based on spatial-semantic alignment as described in claim 7, characterized in that, To ensure the alignment consistency of spatial-semantic features and the robustness of cross-level extraction, a multi-task joint optimization strategy is adopted. The loss function consists of two parts: （9） In the formula, It is a hyperparameter used to balance the alignment loss. and extraction loss Weight in total loss.

9. The cross-modal document information extraction method based on spatial-semantic alignment as described in claim 8, characterized in that, For the alignment loss, used to constrain the matching consistency between spatial and semantic features, cosine similarity is used to calculate the loss as follows: （10）。 10. The cross-modal document information extraction method based on spatial-semantic alignment as described in claim 8, characterized in that, To extract losses, including entity losses and relationship prediction losses; The entity loss is calculated using cross-entropy as follows: （11） In the formula, It is the actual entity label of the i-th text block; The relationship loss also uses cross-entropy loss: （12） Where n is the number of entity pairs, E is the set of entity pairs, and ε is the set of all entity pairs in the relation to be predicted. This indicates that each pair of entities in the set is traversed. The relationship prediction loss for each pair of entities is calculated, and the overall error of relationship extraction is finally measured by the cross-entropy loss function. The true relation labels for entity pairs (i,j) are given, and the final loss is calculated as follows: （13）。

Images