Document information extraction method and apparatus based on image-text modality fusion, and storage medium

Abstract

The present application relates to the technical field of artificial intelligence. Disclosed are a document information extraction method and apparatus based on image-text modality fusion, and a storage medium. The method comprises: acquiring a text block in an image to be processed; extracting a semantic vector from the text block on the basis of a pre-trained NLP model, extracting a vision vector by using computer vision technology, and using an attention mechanism to fuse the semantic vector and the vision vector to form image-text fused feature representation; on the basis of image-text fused features, constructing a fully-connected directed graph; on a constructed graph network, executing graph attention convolution over "node-edge-node" triple feature groups; dynamically weighting salient "text-position-text" triple information by means of a self-attention mechanism; and outputting a feature vector fusing the text, vision and position information. The present disclosure deeply fuses the text, vision and position information, so as to obviously improve the accuracy and robustness of text information matching, thus adapting to complex and changeable image-text environments.

Description

Document information extraction method and device based on image-text modal fusion and storage medium TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to image recognition and natural language processing technology of artificial intelligence, such as a document information extraction method, device and storage medium based on image-text modal fusion. BACKGROUND

[0002] In the field of document intelligence, how to correctly extract structured information from unstructured text has become a hot issue, especially when filling and verifying certificate information, this problem is particularly important. The traditional template matching method performs well in single format certificates, but when facing license certificates with various formats and long duration, template matching encounters difficulties due to differences between different regions. After in-depth study of image characteristics, it is found that in the information matching process, in addition to the text content itself, the position of the text in the image and the visual features of the text also have a significant impact on the matching result.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art.

[0004] SUMMARY

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an overall description of the application, nor is it intended to identify key / important elements or delineate the scope of the embodiments, but to serve as a prelude to the detailed description below.

[0006] The embodiments of the present disclosure provide a document information extraction method, device and storage medium based on image-text modal fusion, which focuses on the triad feature mining of "text-position-text", aiming to improve the discrimination of semantically similar texts, and thus more effectively extract structured information from complex unstructured texts.

[0007] In some embodiments, the method comprises:

[0008] Data processing: obtaining text blocks in the picture to be processed, the text blocks containing their visual features and text content in the picture, the visual features including coordinate information;

[0009] Feature extraction and fusion: extracting semantic vectors from the text blocks according to a pre-trained NLP model, and extracting visual vectors from the text blocks using computer vision technology, and then fusing the semantic vectors and the visual vectors through an attention mechanism to form a feature representation of image-text fusion;

[0010] Graph network construction: According to the characteristics of text and image fusion, a fully connected directed graph is constructed.

[0011] Graph attention convolution: On the constructed graph network, graph attention convolution is performed on the "node-edge-node" ternary feature group.

[0012] Network feature weighting: Through the self-attention mechanism, important "text-location-text" ternary information is dynamically weighted to enhance the expression of key information.

[0013] Output: After graph attention convolution and network feature weighting processing, the feature vector fused with text, visual and location information is output.

[0014] Optionally, in the data processing stage, obtaining the text block in the picture to be processed includes:

[0015] Using an OCR system to process the input picture to extract the text block in the picture;

[0016] Obtaining the detailed coordinates, size and text content of each text block.

[0017] Optionally, in the feature extraction and fusion stage, a self-attention layer is constructed, which accepts semantic vectors and visual vectors as input, and determines the correlation between semantic vectors and visual vectors by calculating the scaled dot product between vectors.

[0018] Optionally, a multi-head self-attention mechanism is used to fuse semantic vectors and visual vectors to dynamically focus on key information in text and images; according to the calculated correlation, the semantic vectors and visual vectors are weighted and adjusted to realize fusion.

[0019] Optionally, in the graph network construction stage, the semantic vectors and visual vectors are fused as the initial features of the nodes in the graph network, and the position embedding vectors of the current text and adjacent texts are calculated as the initial features of the edges in the graph network.

[0020] Optionally, the position embedding vector is generated by a set of predefined sine and cosine functions according to the position index, which is used to capture the position relationship between text blocks; the position embedding vector is combined with the word embedding or character embedding vector to form the complete embedding representation of the model input.

[0021] Optionally, a GAT network is used to construct a graph structure, each node represents a text instance, and the text instance contains fused semantic and visual feature vectors; the graph convolution model updates the feature representation of the node by aggregating and transforming the neighbor information of the node, and performs convolution on the "node-edge-node" ternary feature group, thereby capturing the local and global structure information in the graph to better understand and process the graph data.

[0022] In some embodiments, the apparatus comprises:

[0023] a data processing module configured to obtain a text block in a picture to be processed, the text block containing visual features and text content thereof in the picture, the visual features including coordinate information;

[0024] a feature extraction and fusion module configured to extract a semantic vector from the text block according to a pre-trained NLP model, and extract a visual vector from the text block using a computer vision technology, and then fuse the semantic vector and the visual vector through an attention mechanism to form a feature representation of the image-text fusion;

[0025] a graph network construction module configured to construct a fully connected directed graph according to the image-text fusion feature;

[0026] a graph attention convolution module configured to perform graph attention convolution of a "node-edge-node" triple feature group on the constructed graph network;

[0027] a network feature weighting module configured to dynamically weight important "text-position-text" triple information through a self-attention mechanism to enhance the expression of key information;

[0028] an output module configured to output a feature vector fused with text, visual and position information after graph attention convolution and network feature weighting processing.

[0029] In some embodiments, the apparatus includes a processor and a storage medium having program instructions stored therein, the processor being configured to execute the aforementioned document information extraction method based on image-text modal fusion when the program instructions are run.

[0030] In some embodiments, the storage medium has program instructions stored therein, the program instructions being executed to perform the aforementioned document information extraction method based on image-text modal fusion when run.

[0031] The document information extraction method, apparatus and storage medium based on image-text modal fusion provided by the embodiments of the present disclosure can achieve the following technical effects:

[0032] The present disclosure combines graph network representation and graph attention convolution mechanism, so that the text information matching technology proposed in the present patent has the following significant advantages:

[0033] Information richness: the fusion of text content, visual features and position relationships of the text block provides comprehensive semantic information, which helps to accurately distinguish text with similar semantics.

[0034] Robustness: the graph attention network can adapt to text matching tasks in various complex scenarios and has good robustness to noise, deformation and other interference factors.

[0035] Efficiency: The structure design of the full connection directed graph enables the network to quickly capture the global relationship between text blocks, improving the efficiency of text matching.

[0036] The present disclosure significantly improves the accuracy and robustness of text information matching by deeply fusing text, visual and location information. Compared with traditional methods, the present disclosure is more adaptable to complex and variable image text environments, and brings substantial technical progress to the fields of information retrieval, intelligent question answering, etc.

[0037] The foregoing general description and the following description are only exemplary and explanatory, and are not intended to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0038] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, which are schematic and not intended to be limiting of the embodiments, and in which like reference numerals designate similar items in the figures, and wherein:

[0039] FIG. 1 is a schematic diagram of a document information extraction principle based on image-text modal fusion according to an embodiment of the present disclosure;

[0040] FIG. 2 is a schematic diagram of a document information extraction device based on image-text modal fusion according to an embodiment of the present disclosure;

[0041] FIG. 3 is a schematic diagram of another document information extraction device based on image-text modal fusion according to an embodiment of the present disclosure; DETAILED DESCRIPTION

[0042] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to facilitate the drawings.

[0043] The terms "first", "second", and the like in the embodiments of the present disclosure are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0044] Unless otherwise specified, the term "a plurality of" means two or more.

[0045] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the preceding and following objects. For example, A / B represents: A or B.

[0046] The term "and / or" is a description of the association relationship of objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, three relationships.

[0047] The term "corresponding" can refer to an association relationship or a binding relationship. A and B correspond to each other means that A and B have an association relationship or a binding relationship.

[0048] Information extraction is the process of extracting structured information from unstructured text. Traditional information extraction focuses on extracting entity and relationship information from pure text, but there is less research on visual-rich text. Visual-rich text data refers to semantic structures that are determined not only by the content of the text, but also by the visual features such as the position relationship of the text, font size, etc. There is rich semantic information. Certificates, receipts, invoices, etc. are common visual-rich text data, and most of the information in them is arranged in the form of key-value pairs. Through pre-training, these position information naturally aligned with the text can provide more rich semantic information for the downstream information extraction task.

[0049] Attention mechanism: Attention mechanism is a brain signal processing mechanism unique to human vision. Human vision can quickly scan the global image and obtain the target area that needs to be focused on, which is generally called attention focus, and then invest more attention resources in this area to obtain more detailed information of the target that needs to be focused on, and suppress other useless information. This mechanism can greatly improve the efficiency and accuracy of visual information processing. In the field of deep learning, attention mechanism is introduced to help the model automatically learn and understand the importance of data, so as to allocate more computing resources to important parts.

[0050] Self-attention mechanism: Self-attention mechanism is a special form of attention mechanism, which allows the model to focus on different parts of the sequence when processing a sequence. In self-attention, each input element interacts with all other elements in the sequence and calculates a weighted representation based on these interactions. This mechanism is particularly suitable for processing sequence data, such as text data in natural language processing tasks. Through self-attention, the model can understand the dependency between words in the sequence, regardless of their position in the sequence.

[0051] In simple terms, attention mechanisms are a broader concept that can be applied in various scenarios, including but not limited to sequence-to-sequence tasks. While self-attention mechanisms are a special application of attention mechanisms on sequential data (such as text), they enable the model to focus on the relationships between elements within the sequence. Therefore, while self-attention mechanisms are a type of attention mechanism, they differ in their specific implementation and application.

[0052] The present application is based on the technology of graph attention neural network for image-text modal fusion, which solves the problem of text information matching, especially in the text matching of card and certificate images. The technology aims to improve the accuracy and discrimination of text matching by deeply fusing the textual content, visual features, and positional relationships between texts of image-text blocks with rich semantic information.

[0053] In combination with FIG. 1, the present disclosure provides a document information extraction device based on image-text modal fusion, which comprises:

[0054] 1. Data processing module:

[0055] The input image is scanned using OCR (Optical Character Recognition) technology to accurately extract text blocks.

[0056] The detailed coordinates, dimensions, and text content of each text block are obtained.

[0057] 2. Feature extraction and fusion module:

[0058] A pre-trained NLP (Natural Language Processing) model is used to convert the text content into semantic vectors rich in semantics.

[0059] Computer vision techniques are used to extract visual features from the text blocks and convert them into visual vectors.

[0060] The semantic vectors and visual vectors are fused through an attention mechanism to form a feature representation of image-text fusion.

[0061] 3. Graph network construction module:

[0062] A fully connected directed graph is constructed with the image-text fusion features as node features.

[0063] The edge features between nodes are defined by the position embedding vectors between text blocks, which are generated through predefined sine and cosine functions to capture the positional relationships between texts.

[0064] 4. Graph attention convolution module:

[0065] On the constructed graph network, graph attention convolution of the "node-edge-node" triple feature group is performed.

[0066] 5. Network feature weighting module:

[0067] The self-attention mechanism is introduced to dynamically weight important "text-location-text" triplets, enhancing the expression of key information.

[0068] 6. Output module:

[0069] After processing by the graph attention network, a feature vector that integrates text, visual, and location information is output.

[0070] In the text information matching task, high-precision text information matching is achieved by calculating the similarity between these feature vectors.

[0071] It can be understood that the fusion of semantic vectors and visual vectors based on the self-attention mechanism is an effective method that allows the model to dynamically focus on key information in different modalities (text and image) during the fusion process. First, the text information in the image is extracted through OCR technology, which is used to convert the text in the image into editable and searchable text.

[0072] Since OCR itself does not provide semantic vectors for the text, a pre-trained NLP model is needed to convert the text into a vector representation rich in semantic information. At the same time, computer vision technology is used to extract features from the image within the text coordinates and convert them into corresponding visual vectors.

[0073] Next, a self-attention layer is constructed, which accepts semantic vectors and visual vectors as input and determines their correlation by calculating the scaled dot product between vectors. This mechanism not only considers intra-modal relationships (i.e., context within text or within image), but also considers inter-modal relationships (i.e., interaction between text and image).

[0074] By adopting a multi-head self-attention design, different heads can focus on different modalities or combinations of modalities. Finally, according to the calculated correlation, the semantic vectors and visual vectors are weighted and adjusted, and fusion is achieved through weighted summation or weighted splicing, etc. During this process, the fused vectors not only integrate information from different modalities, but also balance according to their respective weights obtained in the self-attention mechanism, ensuring that the model can dynamically focus on key information in text and image when making decisions.

[0075] Among them, the position embedding uses a set of predefined sine and cosine functions to generate position embeddings according to the index of each position. The embedding of each position is calculated by points on sine and cosine waves of different frequencies. A key feature of this method is that it does not depend on the model training process, and the embedding is fixed and the same for all input sequences. Through position embedding, the model can capture the positional relationship between words or characters, which is crucial for understanding the structure and semantics of sentences.

[0076] Position embeddings can help the model distinguish such relationships, leading to a more accurate understanding of the sentence's meaning. Position information is mapped to a continuous vector space, resulting in position embedding vectors. These vectors, combined with word or character embedding vectors, form the complete embedding representation of the model input.

[0077] A GAT network is used to construct the graph structure, with each node representing a text instance and its features being the fused semantic and visual feature vectors. The initial features of the edges are defined by the position embedding vectors. The graph convolution model updates the node's feature representation by aggregating and transforming the neighbor information of the node, and performs convolution on the triple feature group of "node-edge-node". This process helps the model capture local and global structure information in the graph, leading to a better understanding and processing of graph data.

[0078] As an example, the present disclosure provides a document information extraction method based on graph-text modal fusion as follows:

[0079] (1) Data processing

[0080] An OCR (Optical Character Recognition) system is used to process each input image and extract text blocks from the image. Each text block contains its coordinate information in the image (such as the coordinates of the top-left corner and the bottom-right corner), visual features such as length and width, and text content.

[0081] (2) Feature fusion

[0082] According to the OCR results, semantic vectors are extracted using NLP models, and visual vectors are extracted using visual computing techniques. An attention mechanism is introduced to fuse the two vectors as the initial features of the nodes. The position embedding vectors of the current text and adjacent texts are calculated as the initial features of the edges.

[0083] (3) Graph network construction

[0084] Using graph-text multi-modal feature fusion as the initial feature of the node and position embedding vector as the initial feature of the edge, a graph network is constructed to form a fully connected directed graph.

[0085] (4) Triple feature group convolution

[0086] On the constructed fully connected directed graph, graph attention convolution operations are performed on the triple feature group of "node-edge-node". The convolution operation can consider the node features, edge features, and adjacent node features to extract more rich fusion features.

[0087] (5) Network feature weighting

[0088] The graph network introduces a self-attention mechanism, calculates the attention weight of each triple, and aggregates the features by weighting, so that the network can pay attention to more important "text-position-text" triple information.

[0089] (6) Feature fusion and output

[0090] After the convolution and self-attention mechanism of the graph attention network, each node obtains a feature vector that integrates the text content, visual features, and position relationship. These feature vectors not only contain semantic information of the text block, but also reflect the spatial relationship between the text blocks.

[0091] In the text information matching task, the similarity between the feature vectors of different text blocks in the picture is calculated to determine the matching degree of the text information.

[0092] The present disclosure proposes a document information extraction method and device based on graph-text modal fusion to solve the problem of text information matching. After passing through the OCR system, each picture will obtain a set of text blocks, each of which contains its coordinate information and text content in the picture. The graph attention network is used to model the text content and visual features in the text block and the position relationship between the texts, and the fusion of the features enriches the semantic information of the card certificate image, further improving the discrimination degree of the semantically similar texts.

[0093] This work uses the text blocks obtained by OCR to form a fully connected directed graph, each text block constituting a node, and each node being connected to all other nodes. The initial features of the nodes are composed of the text content and visual features of the text blocks. The initial features of the edges are composed of the coordinates, length and width of the current text block, and the position relationship with the adjacent text blocks.

[0094] Secondly, the network performs convolution on the triple feature group of "node-edge-node", and the self-attention mechanism in it allows the network to select more noticeable information from all directed triples formed by the fully connected directed graph and aggregate the features by weighting. Pay more attention to the correlation degree of the "text-position-text" triple information, and effectively distinguish the texts.

[0095] The document information extraction technology proposed by the present disclosure has the following significant advantages:

[0096] Information richness: The fusion of the text content, visual features, and position relationship of the text blocks provides comprehensive semantic information, which helps to accurately distinguish semantically similar texts.

[0097] Robustness: The graph attention network can adapt to various text matching tasks in complex scenarios and has good robustness to noise, deformation, and other interference factors.

[0098] Efficiency: The structure design of the full connection directed graph enables the network to quickly capture the global relationship between text blocks, improving the efficiency of text matching.

[0099] The present disclosure significantly improves the accuracy and robustness of text information matching by deeply fusing text, visual and location information. Compared with traditional methods, the present disclosure is more adaptable to complex and variable image text environments, and brings substantial technical progress to the fields of information retrieval, intelligent question answering, etc.

[0100] In combination with FIG. 2, the present disclosure provides a document information extraction device 200 based on graph-text modal fusion, which comprises a data processing module 201, a feature extraction and fusion module 202, a graph network construction module 203, a graph attention convolution module 204, a network feature weighting module 205 and an output module 206.

[0101] The data processing module 201 is configured to obtain text blocks in a picture to be processed, the text blocks containing their visual features and text content in the picture, the visual features including coordinate information;

[0102] The feature extraction and fusion module 202 is configured to extract semantic vectors from the text blocks according to a pre-trained NLP model, and extract visual vectors from the text blocks using computer vision technology, and then fuse the semantic vectors and the visual vectors through an attention mechanism to form a graph-text fusion feature representation;

[0103] The graph network construction module 203 is configured to construct a full connection directed graph according to the graph-text fusion features;

[0104] The graph attention convolution module 204 is configured to perform graph attention convolution of a “node-edge-node” triple feature group on the constructed graph network;

[0105] The network feature weighting module 205 is configured to dynamically weight important “text-location-text” triple information through a self-attention mechanism to enhance the expression of key information;

[0106] The output module 206 is configured to output a feature vector fused with text, visual and location information after graph attention convolution and network feature weighting processing.

[0107] With reference to FIG. 3, the disclosure provides a document information extraction device 300 based on image-text modal fusion, which comprises a processor 304 and a memory 301. Optionally, the device can further comprise a communication interface 302 and a bus 303. The processor 304, the communication interface 302 and the memory 301 can communicate with each other through the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can invoke the logical instructions in the memory 301 to execute the method for… of the above-mentioned embodiments.

[0108] In addition, the logical instructions in the memory 301 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0109] The memory 301 as a computer readable storage medium can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method of the embodiments of the disclosure. The processor 304 executes the program instructions / modules stored in the memory 301 to perform functional applications and data processing, i.e. to implement the method of the above-mentioned embodiments based on image-text modal fusion of document information extraction.

[0110] The memory 301 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 301 can include a high-speed random access memory, and can also include a non-volatile memory.

[0111] The disclosure provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are configured to execute the above-mentioned method of document information extraction based on image-text modal fusion.

[0112] The above-mentioned computer readable storage medium can be a transitory computer readable storage medium or a non-transitory computer readable storage medium.

[0113] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method disclosed in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes, and can also be a transitory storage medium.

[0114] The above description and drawings sufficiently illustrate the embodiments of the present disclosure to enable one skilled in the art to practice them. Other embodiments can include structural, logical, electrical, process, and other changes. The embodiments represent only a few of the possible variations. Individual components and functions are optional unless explicitly required, and the order of operations can be changed. Parts and features of some embodiments can be included in or replace parts and features of other embodiments. Also, the words used in this application are only used to describe the embodiments and not to limit the scope of protection. As used in the description herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations of one or more associated listed items. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising" and the like mean the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, or device that includes the stated element. In this document, each embodiment focuses on the differences from other embodiments, and the same or similar parts between various embodiments can be referred to each other. For the method, product, etc. disclosed in the embodiments, if it corresponds to the method part disclosed in the embodiments, the relevant part can be referred to the description of the method part.

[0115] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present disclosure. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0116] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units can only be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to implement the embodiments. In addition, each functional unit in the embodiments of the present disclosure can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

[0117] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

Claims

1. A document information extraction method based on image-text modal fusion, characterized in that, Comprising: Data processing: obtaining text blocks in the picture to be processed, the text blocks containing their visual features in the picture and text content, the visual features including coordinate information; Feature extraction and fusion: extracting semantic vectors from the text blocks according to a pre-trained NLP model, and extracting visual vectors from the text blocks using computer vision technology, and then fusing the semantic vectors and the visual vectors through an attention mechanism to form a feature representation of the fusion of text and image; Graph network construction: constructing a fully connected directed graph according to the fusion feature of text and image; Graph attention convolution: performing graph attention convolution on the "node-edge-node" ternary feature group on the constructed graph network; Network feature weighting: dynamically weighting important "text-position-text" ternary information through a self-attention mechanism to enhance the expression of key information; Output: outputting a feature vector that fuses text, visual and position information after graph attention convolution and network feature weighting processing.

2. The method of claim 1, wherein, In the data processing stage, the obtaining of the text blocks in the picture to be processed comprises: processing the input picture using an OCR system to extract the text blocks in the picture; obtaining detailed coordinates, dimensions and text content of each text block.

3. The method of claim 1, wherein, In the feature extraction and fusion stage, a self-attention layer is constructed, which accepts semantic vectors and visual vectors as input and determines the correlation between semantic vectors and visual vectors by calculating the scaled dot product between vectors.

4. The method of claim 3, wherein a multi-head self-attention mechanism is used to fuse semantic vectors and visual vectors to dynamically focus on key information in text and image; the semantic vectors and the visual vectors are weighted and adjusted according to the calculated correlation to achieve fusion.

5. The method according to any one of claims 1 to 4, characterized in that, In the graph network construction stage, the semantic vectors and the visual vectors are fused as the initial features of the nodes in the graph network, and the position embedding vectors of the current text and the adjacent text are calculated as the initial features of the edges in the graph network.

6. The method of claim 5, wherein the position embedding vectors are generated by a set of predefined sine and cosine functions according to the position index to capture the position relationship between text blocks; the position embedding vectors are combined with word embedding or character embedding vectors to form a complete embedding representation of the model input.

7. The method of claim 5, wherein a GAT network is used to construct a graph structure, each node representing a text instance containing fused semantic and visual feature vectors; the graph convolution model updates the feature representation of the node by aggregating and transforming the neighbor information of the node, and performs convolution on the "node-edge-node" ternary feature group, thereby capturing local and global structure information in the graph to better understand and process graph data.

8. A document information extraction apparatus based on image-text modality fusion, characterized by, Comprising: a data processing module configured to obtain text blocks in the picture to be processed, the text blocks containing their visual features in the picture and text content, the visual features including coordinate information; The feature extraction and fusion module is configured to extract semantic vectors from the text block according to a pre-trained NLP model, extract visual vectors from the text block using computer vision technology, and then fuse the semantic vectors and the visual vectors through an attention mechanism to form a feature representation of the image-text fusion. The graph network construction module is configured to construct a fully connected directed graph according to the image-text fusion feature. The graph attention convolution module is configured to perform graph attention convolution of a "node-edge-node" triple feature group on the constructed graph network. The network feature weighting module is configured to dynamically weight important "text-position-text" triple information through a self-attention mechanism to enhance the expression of key information. The output module is configured to output a feature vector that fuses text, visual and position information after graph attention convolution and network feature weighting processing.

9. A document information extraction device based on image-text modality fusion, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute the document information extraction method based on image-text modal fusion as claimed in any one of claims 1 to 7 when running the program instructions.

10. A storage medium storing program instructions, characterized in that, The program instructions, when executed, perform the document information extraction method based on image-text modal fusion as claimed in any one of claims 1 to 7.

Images