Document information extraction method and apparatus based on image-text modal fusion, and storage medium

Abstract

A document information extraction method and apparatus based on image-text modal fusion, and a storage medium, relating to the technical field of artificial intelligence. The method comprises: acquiring a text block in a picture to be processed; extracting a semantic vector from the text block on the basis of a pre-trained NLP model, extracting a visual vector by using computer vision technology, and fusing the semantic vector and the visual vector by means of an attention mechanism to form an image-text fused feature representation; constructing a fully-connected directed graph on the basis of image-text fused feature; on the constructed graph network, performing graph attention convolution on "node-edge-node" triplet feature sets; dynamically weighting important "text-position-text" triplet information by means of a self-attention mechanism; and outputting a feature vector in which text, vision and position information are fused. By deep fusion of text, vision and position information, the present disclosure significantly improves the accuracy and robustness of text information matching, thus adapting to complex and diverse image-text environments.

Description

Document information extraction method and device based on image-text modal fusion and storage medium

[0001] The present application claims priority to the Chinese patent application No. CN202410625846.8, filed on May 20, 2024, and entitled "Document information extraction method and device based on image-text modal fusion and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, in particular to image recognition and natural language processing technology of artificial intelligence, for example, a document information extraction method and device based on image-text modal fusion and storage medium. BACKGROUND

[0003] In the field of document intelligence, how to correctly extract structured information from unstructured text has become a hot issue, especially when filling and verifying certificate information, this problem is particularly important. Traditional template matching-based methods perform well in single-format certificates, however, when facing license certificates with diverse formats and long duration, such as business licenses, template matching encounters difficulties due to differences between different regions. After in-depth study of image characteristics, it is found that in the information matching process, in addition to the text content itself, the position of the text in the image and the visual features of the text also have a significant impact on the matching results.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an overall description of the application, nor is it intended to identify key / important elements or delineate the scope of the embodiments, but to serve as a prelude to the detailed description below.

[0006] The embodiments of the present disclosure provide a document information extraction method and device based on image-text modal fusion and storage medium, which focuses on the triad feature mining of "text-position-text", aiming to improve the discrimination of semantically similar texts, and thus more effectively extract structured information from complex unstructured texts.

[0007] In some embodiments, the method comprises:

[0008] Data processing: obtaining text blocks in the picture to be processed, the text blocks containing their visual features and text content in the picture, the visual features including coordinate information;

[0009] Feature extraction and fusion: semantic vectors are extracted from the text blocks according to a pre-trained NLP (Netural Language Processing) model, and visual vectors are extracted from the text blocks using computer vision technology, and then the semantic vectors and the visual vectors are fused through an attention mechanism to form a feature representation of the fusion of text and image;

[0010] Graph network construction: a fully connected directed graph is constructed according to the fusion feature of text and image;

[0011] Graph attention convolution: on the constructed graph network, graph attention convolution of a "node-edge-node" ternary feature group is performed;

[0012] Network feature weighting: through a self-attention mechanism, important "text-position-text" ternary information is dynamically weighted to enhance the expression of key information;

[0013] Output: after the graph attention convolution and the network feature weighting processing, a feature vector that fuses text, visual and position information is output.

[0014] Optionally, in the data processing stage, obtaining the text blocks in the picture to be processed includes:

[0015] processing the input picture using an OCR (Optical Character Recognition) system to extract the text blocks in the picture;

[0016] obtaining detailed coordinates, sizes and text content of each text block.

[0017] Optionally, in the feature extraction and fusion stage, a self-attention layer is constructed, which receives the semantic vectors and the visual vectors as inputs and determines the correlation degree between the semantic vectors and the visual vectors by calculating the scaled dot product between the vectors.

[0018] Optionally, a multi-head self-attention mechanism is used to fuse the semantic vectors and the visual vectors to dynamically focus on key information in the text and the image; and the semantic vectors and the visual vectors are weighted and adjusted according to the calculated correlation degree to realize fusion.

[0019] Optionally, in the graph network construction stage, the semantic vectors and the visual vectors are fused as the initial features of the nodes in the graph network, and the position embedding vectors of the current text and the adjacent texts are calculated as the initial features of the edges in the graph network.

[0020] Optionally, the position embedding vectors are generated according to the position index through a set of predefined sine and cosine functions to capture the position relationship between the text blocks; the position embedding vectors are combined with the word embedding or character embedding vectors to form a complete embedding representation of the model input.

[0021] Optionally, a GAT (Graph Attention Network) network is used to construct a graph structure, each node representing a text instance, the text instance containing fused semantic and visual feature vectors; a graph convolution model updates the feature representation of a node by aggregating and transforming the neighbor information of the node, and performs convolution on a ternary feature group of "node-edge-node", so as to capture local and global structure information in the graph, and better understand and process the graph data.

[0022] In some embodiments, the apparatus comprises:

[0023] a data processing module configured to obtain a text block in a picture to be processed, the text block containing visual features and text content thereof in the picture, the visual features including coordinate information;

[0024] a feature extraction and fusion module configured to extract a semantic vector from the text block according to a pre-trained NLP model, and extract a visual vector from the text block using computer vision technology, and then fuse the semantic vector and the visual vector through an attention mechanism to form a feature representation of the fusion of the text and the picture;

[0025] a graph network construction module configured to construct a fully connected directed graph according to the fusion feature of the text and the picture;

[0026] a graph attention convolution module configured to perform graph attention convolution of a ternary feature group of "node-edge-node" on the constructed graph network;

[0027] a network feature weighting module configured to dynamically weight important "text-position-text" ternary information through a self-attention mechanism to enhance the expression of key information;

[0028] an output module configured to output a feature vector fused with text, visual and position information after graph attention convolution and network feature weighting processing.

[0029] In some embodiments, the apparatus comprises a processor and a memory storing program instructions, the processor being configured to execute the aforementioned document information extraction method based on fusion of text and picture modalities when running the program instructions.

[0030] In some embodiments, the storage medium stores program instructions, the program instructions being executed to perform the aforementioned document information extraction method based on fusion of text and picture modalities.

[0031] The document information extraction method, apparatus and storage medium based on fusion of text and picture modalities provided by the embodiments of the present disclosure can achieve the following technical effects:

[0032] The present disclosure combines graph network representation and graph attention convolution mechanism, so that the text information matching technology proposed by the present disclosure has the following advantages:

[0033] Information richness: The fusion of text block content, visual features and location relationship provides comprehensive semantic information, which helps to accurately distinguish similar semantic texts.

[0034] Robustness: The graph attention network can adapt to various complex scene text matching tasks and has good robustness to noise, deformation and other interference factors.

[0035] Efficiency: The structure design of the full connection directed graph enables the network to quickly capture the global relationship between text blocks, improving the efficiency of text matching.

[0036] The present disclosure significantly improves the accuracy and robustness of text information matching by deeply fusing text, visual and location information. Compared with traditional methods, the present disclosure is more suitable for complex and variable image text environment, and brings substantial technical progress for information retrieval, intelligent question answering and other fields.

[0037] The foregoing general description and the following description are only exemplary and explanatory, and are not intended to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0038] One or more embodiments are exemplarily illustrated by corresponding drawings, which do not constitute limitation on the embodiments, elements with the same reference numerals in the drawings are shown as similar elements, the drawings do not constitute proportional limitation, and wherein:

[0039] Fig. 1 is a schematic diagram of a document information extraction principle based on graph-text modal fusion provided by an embodiment of the present disclosure;

[0040] Fig. 2 is a schematic diagram of a document information extraction device based on graph-text modal fusion provided by an embodiment of the present disclosure;

[0041] Fig. 3 is a schematic diagram of another document information extraction device based on graph-text modal fusion provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In order to be able to understand the features and technical contents of the embodiments of the present disclosure more fully, the implementation of the embodiments of the present disclosure will be described in detail below, and the attached drawings are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to show.

[0043] The terms "first", "second", etc. in the embodiments of the present disclosure are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so as to implement the embodiments of the present disclosure described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0044] Unless otherwise specified, the term "a plurality of" means two or more.

[0045] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B represents: A or B.

[0046] The term "and / or" is a description of the association relationship between objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, the three relationships.

[0047] The term "corresponding" can refer to an association or binding relationship. A and B correspond to each other means that there is an association or binding relationship between A and B.

[0048] Information extraction is the process of extracting structured information from unstructured text. Traditional information extraction focuses on extracting entity and relationship information from pure text, but there is less research on visual-rich text. Visual-rich text data refers to semantic structures that are determined not only by the content of the text, but also by the visual features of the text position relationship, font size, etc. There is rich semantic information. Certificates, receipts, invoices, etc. are common visual-rich text data, and most of the information in them is arranged in the form of key-value pairs. Through pre-training, these naturally aligned position information with the text can provide more rich semantic information for the downstream information extraction task.

[0049] Attention mechanism: Attention mechanism is a brain signal processing mechanism unique to human vision. Human vision can quickly scan the global image and obtain the target area that needs to be focused on, which is generally called attention focus. Then more attention resources are invested in this area to obtain more detailed information of the target that needs to be focused on, and other useless information is suppressed. This mechanism can greatly improve the efficiency and accuracy of visual information processing. In the field of deep learning, attention mechanism is introduced to help the model automatically learn and understand the importance of data, so as to allocate more computing resources to important parts.

[0050] Self-Attention Mechanism: Self-Attention Mechanism is a special form of Attention Mechanism that allows the model to focus on different parts of the sequence when processing a sequence. In Self-Attention, each input element interacts with all other elements in the sequence and computes a weighted representation based on these interactions. This mechanism is particularly suitable for handling sequence data, such as text data in natural language processing tasks. Through Self-Attention, the model can understand the dependencies between words in a sequence, regardless of their position in the sequence.

[0051] In simple terms, Attention Mechanism is a broader concept that can be applied to various scenarios, including but not limited to sequence-to-sequence tasks. While Self-Attention Mechanism is a special application of Attention Mechanism on sequence data (such as text), it enables the model to focus on the relationships between elements within a sequence. Therefore, although Self-Attention Mechanism is a type of Attention Mechanism, they differ in their specific implementation and application.

[0052] Based on the graph-text modal fusion graph attention neural network technology, the application solves the problem of text information matching, especially in the text matching of card and certificate images. The technology aims to improve the accuracy and discrimination of text matching by deeply fusing the textual content, visual features, and positional relationships between texts of image and text blocks, with rich semantic information.

[0053] In combination with FIG. 1, the present disclosure provides a document information extraction device based on graph-text modal fusion, which comprises:

[0054] 1. Data processing module: use OCR (Optical Character Recognition) technology to scan the input image and accurately extract the text block.

[0055] Obtain the detailed coordinates, size, and text content of each text block.

[0056] 2. Feature extraction and fusion module:

[0057] Use a pre-trained NLP (Natural Language Processing) model to convert the text content into semantic vectors rich in semantics.

[0058] Use computer vision technology to extract visual features from the text block and convert them into visual vectors.

[0059] Fuse the semantic vectors and visual vectors through the attention mechanism to form a graph-text fusion feature representation.

[0060] 3. Graph network construction module:

[0061] Construct a fully connected directed graph with graph-text fusion features as node features.

[0062] The edge feature between nodes is defined by the position embedding vector between text blocks, which is generated by a predefined sine and cosine function to capture the positional relationship between texts.

[0063] 4. Graph attention convolution module:

[0064] On the constructed graph network, graph attention convolution of the "node-edge-node" triplet feature group is performed.

[0065] 5. Network feature weighting module:

[0066] The self-attention mechanism is introduced to dynamically weight important "text-position-text" triplet information and enhance the expression of key information.

[0067] 6. Output module:

[0068] After processing by the graph attention network, the feature vector that integrates text, visual, and position information is output.

[0069] In the text information matching task, high-precision text information matching is achieved by calculating the similarity between these feature vectors.

[0070] It can be understood that the fusion of semantic vectors and visual vectors based on the self-attention mechanism is an effective method that allows the model to dynamically focus on key information in different modalities (text and image) during fusion. First, the text information in the image is extracted by OCR technology, which is used to convert the text in the image into editable and searchable text.

[0071] Since OCR itself does not provide semantic vectors for the text, a pre-trained NLP model is needed to convert the text into a vector representation rich in semantic information. At the same time, computer vision technology is used to extract features from the image within the text coordinates and convert them into corresponding visual vectors.

[0072] Next, a self-attention layer is constructed, which receives semantic vectors and visual vectors as input and determines their correlation by calculating the scaled dot product between vectors. This mechanism not only considers intra-modal relationships (i.e., context within text or within image), but also considers inter-modal relationships (i.e., interaction between text and image).

[0073] By adopting the multi-head self-attention design, different heads can focus on different modalities or combinations of modalities. Finally, according to the calculated correlation degree, the semantic vector and the visual vector are weighted and adjusted, and the fusion is realized through weighted summation or weighted splicing, etc. In this process, the fused vector not only integrates information from different modalities, but also balances according to the weight obtained in the self-attention mechanism, so as to ensure that the model can dynamically focus on the key information in the text and image when making decisions.

[0074] where the position embedding uses a set of predefined sine and cosine functions to generate position embeddings based on the index of each position. The embedding of each position is calculated by points on sine and cosine waves of different frequencies. One key feature of this method is that it does not rely on the model training process, and the embedding is fixed and the same for all input sequences. Through position embedding, the model can capture the positional relationship between words or characters, which is crucial for understanding the structure and semantics of sentences.

[0075] Position embedding can help the model distinguish such relationships and better understand the meaning of the sentence. Position information is mapped to a continuous vector space to generate position embedding vectors. These vectors, combined with word embedding or character embedding vectors, form the complete embedding representation of the model input.

[0076] GAT network is used to construct a graph structure, where each node represents a text instance, and its feature is the fused semantic and visual feature vector. The initial feature of the edge is defined by the position embedding vector. The graph convolution model updates the feature representation of the node by aggregating and transforming the neighbor information of the node, and performs convolution on the ternary feature group of "node-edge-node". This process helps the model capture local and global structure information in the graph, so as to better understand and process graph data.

[0077] As an example, the present disclosure provides a document information extraction method based on image-text modality fusion as follows:

[0078] (1) Data processing

[0079] An OCR (Optical Character Recognition) system is used to process each input image and extract text blocks from the image. Each text block contains its coordinate information in the image (such as the coordinates of the top-left corner and the bottom-right corner), visual features such as length and width, and text content.

[0080] (2) Feature fusion

[0081] According to the OCR result, the semantic vector is extracted by using the NLP model, and the visual vector is extracted by using the visual computer technology. The attention mechanism is introduced to fuse the two vectors as the initial features of the nodes. The position embedding vector of the current text and the adjacent text is calculated as the initial feature of the edge.

[0082] (3) Graph network construction

[0083] The graph network is constructed by using the multi-modal feature fusion of text and image as the initial feature of the node and the position embedding vector as the initial feature of the edge, and a fully connected directed graph is formed.

[0084] (4) Triplet feature group convolution

[0085] On the constructed fully connected directed graph, the graph attention convolution operation of the triplet feature group of “node-edge-node” is performed. The convolution operation can comprehensively consider the node features, edge features and adjacent node features, so as to extract more rich fusion features.

[0086] (5) Network feature weighting

[0087] The graph network introduces the self-attention mechanism, calculates the attention weight of each triplet, and aggregates the features by weighting, so that the network can pay attention to more important “text-position-text” triplet information.

[0088] (6) Feature fusion and output

[0089] After the convolution of the graph attention network and the processing of the self-attention mechanism, each node will obtain a feature vector that fuses the text content, visual features and position relationship. These feature vectors contain not only the semantic information of the text block, but also the spatial relationship between the text blocks.

[0090] In the text information matching task, the similarity between the feature vectors of different text blocks in the picture is calculated, so as to judge the matching degree of the text information.

[0091] The present disclosure proposes a document information extraction method based on text-image modal fusion, which is used to solve the problem of text information matching. After each picture passes through the OCR system, a group of text blocks will be obtained, each of which contains the coordinate information and text content in the picture. The graph attention network is used to model the text content and visual features in the text block and the position relationship between the texts, and the fusion features enrich the semantic information of the card certificate image, further improving the discrimination degree of the texts with similar semantics.

[0092] The present disclosure utilizes the text blocks obtained by OCR to form a fully connected directed graph, each text block constitutes a node, and each node is connected with all other nodes. The initial features of the nodes are composed of the text content and visual features of the text blocks. The initial features of the edges are composed of the current text block coordinates, length and width, and the positional relationship with adjacent text blocks.

[0093] Secondly, the network performs convolution on the ternary feature group of "node-edge-node", and the self-attention mechanism in it allows the network to select more noteworthy information in all directed ternary groups formed by the fully connected directed graph and aggregate the features with weights. Pay more attention to the correlation degree of the ternary information of 'text-position-text', and effectively distinguish the text.

[0094] The document information extraction technology proposed by the present disclosure has the following significant advantages:

[0095] Information richness: The fusion of text content, visual features and positional relationship of text blocks provides comprehensive semantic information, which helps to accurately distinguish semantically similar texts.

[0096] Robustness: The graph attention network can adapt to various text matching tasks in complex scenarios and has good robustness to noise, deformation and other interference factors.

[0097] Efficiency: The structure design of the fully connected directed graph enables the network to quickly capture the global relationship between text blocks, improving the efficiency of text matching.

[0098] The present disclosure significantly improves the accuracy and robustness of text information matching by deeply fusing text, visual and positional information. Compared with traditional methods, the present invention is more adaptable to complex and variable image text environments, and brings substantial technical progress to the fields of information retrieval and intelligent question answering.

[0099] In combination with FIG. 2, the present disclosure provides a document information extraction device 300 based on graph-text modal fusion, which comprises a data processing module 201, a feature extraction and fusion module 202, a graph network construction module 203, a graph attention convolution module 204, a network feature weighting module 205 and an output module 206.

[0100] The data processing module 201 is configured to obtain text blocks in a picture to be processed, the text blocks containing their visual features and text content in the picture, the visual features including coordinate information;

[0101] The feature extraction and fusion module 202 is configured to extract semantic vectors from the text blocks according to a pre-trained NLP model, and extract visual vectors from the text blocks using computer vision technology, and then fuse the semantic vectors and the visual vectors through an attention mechanism to form a graph-text fusion feature representation;

[0102] The graph network construction module 203 is configured to construct a fully connected directed graph according to the graph-text fusion features.

[0103] The graph attention convolution module 204 is configured to perform graph attention convolution of the "node-edge-node" triple feature group on the constructed graph network.

[0104] The network feature weighting module 205 is configured to dynamically weight important "text-position-text" triple information through a self-attention mechanism to enhance the expression of key information.

[0105] The output module 206 is configured to output a feature vector that fuses text, visual, and position information after graph attention convolution and network feature weighting processing.

[0106] In combination with FIG. 3, the embodiment of the present disclosure provides a document information extraction device 300 based on graph-text modal fusion, which includes a processor 304 and a memory 301. Optionally, the device can also include a communication interface 302 and a bus 303. The processor 304, the communication interface 302, and the memory 301 can communicate with each other through the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can invoke the logical instructions in the memory 301 to execute the document information extraction method based on graph-text modal fusion of the above-mentioned embodiments.

[0107] In addition, the logical instructions in the memory 301 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0108] The memory 301 as a kind of computer readable storage medium can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 304 executes the program instructions / modules stored in the memory 301, thereby executing functional applications and data processing, that is, implementing the method of document information extraction based on graph-text modal fusion in the above-mentioned embodiments.

[0109] The memory 301 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 301 can include a high-speed random access memory, and can also include a non-volatile memory.

[0110] The embodiment of the present disclosure provides a computer readable storage medium, which stores computer executable instructions configured to execute the above-mentioned document information extraction method based on image-text modal fusion.

[0111] The computer readable storage medium described above can be a transitory computer readable storage medium or a non-transitory computer readable storage medium.

[0112] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, including a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes, or a transitory storage medium.

[0113] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for descriptive purposes only and is not intended to limit the scope of protection. As used in the description herein, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used herein means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0115] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to apparatuses, devices, etc.) can be implemented in other manners. For example, the apparatus embodiments described above are merely schematic. For example, the division of the units is merely logical function division. There can be other division manners in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, apparatuses, or units, and can be in electrical, mechanical, or other forms. The units described as separated components can or can not be physically separated, and components displayed as units can or can not be physical units. Some or all of the units can be selected according to actual needs to achieve the embodiments.

[0116] The flowcharts and block diagrams in the drawings show the possible implementation architectures, functions, and operations of the systems, methods, and computer program products according to the embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions noted in the blocks can occur in different orders than those noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the drawings, the operations or steps corresponding to different blocks can also occur in different orders than those disclosed in the descriptions, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. Each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A document information extraction method based on image-text modal fusion, characterized in that, Comprising: Data processing: obtaining text blocks in the picture to be processed, the text blocks containing their visual features in the picture and text content, the visual features including coordinate information; Feature extraction and fusion: extracting semantic vectors from the text blocks according to a pre-trained NLP model, and extracting visual vectors from the text blocks using computer vision technology, and then fusing the semantic vectors and the visual vectors through an attention mechanism to form a feature representation of the fusion of text and image; Graph network construction: constructing a fully connected directed graph according to the fusion feature of text and image; Graph attention convolution: performing graph attention convolution on the "node-edge-node" ternary feature group on the constructed graph network; Network feature weighting: dynamically weighting important "text-position-text" ternary information through a self-attention mechanism to enhance the expression of key information; Output: outputting a feature vector that fuses text, visual and position information after graph attention convolution and network feature weighting processing.

2. The method of claim 1, wherein, In the data processing stage, the obtaining of the text blocks in the picture to be processed comprises: processing the input picture using an OCR system to extract the text blocks in the picture; obtaining detailed coordinates, dimensions and text content of each text block.

3. The method of claim 1, wherein, In the feature extraction and fusion stage, a self-attention layer is constructed, which receives semantic vectors and visual vectors as input and determines the correlation between semantic vectors and visual vectors by calculating the scaled dot product between vectors.

4. The method of claim 3, wherein a multi-head self-attention mechanism is used to fuse semantic vectors and visual vectors to dynamically focus on key information in text and image; the semantic vectors and the visual vectors are weighted and adjusted according to the calculated correlation to achieve fusion.

5. The method according to any one of claims 1 to 4, characterized in that, In the graph network construction stage, the semantic vectors and the visual vectors are fused as the initial features of the nodes in the graph network, and the position embedding vectors of the current text and the adjacent text are calculated as the initial features of the edges in the graph network.

6. The method of claim 5, wherein the position embedding vectors are generated by a set of predefined sine and cosine functions according to the position index to capture the position relationship between text blocks; the position embedding vectors are combined with word embedding or character embedding vectors to form a complete embedding representation of the model input.

7. The method of claim 5, wherein a GAT network is used to construct a graph structure, each node representing a text instance containing fused semantic and visual feature vectors; the graph convolution model updates the feature representation of the node by aggregating and transforming the neighbor information of the node, and performs convolution on the "node-edge-node" ternary feature group, thereby capturing local and global structure information in the graph to better understand and process graph data.

8. A document information extraction apparatus based on image-text modality fusion, characterized by, Comprising: a data processing module configured to obtain text blocks in the picture to be processed, the text blocks containing their visual features in the picture and text content, the visual features including coordinate information; The feature extraction and fusion module is configured to extract semantic vectors from the text block according to a pre-trained NLP model, extract visual vectors from the text block using computer vision technology, and then fuse the semantic vectors and the visual vectors through an attention mechanism to form a feature representation of the image-text fusion. The graph network construction module is configured to construct a fully connected directed graph according to the image-text fusion feature. The graph attention convolution module is configured to perform graph attention convolution of a "node-edge-node" triple feature group on the constructed graph network. The network feature weighting module is configured to dynamically weight important "text-position-text" triple information through a self-attention mechanism to enhance the expression of key information. The output module is configured to output a feature vector that fuses text, visual and position information after graph attention convolution and network feature weighting processing.

9. A document information extraction device based on image-text modality fusion, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute the document information extraction method based on image-text modal fusion as claimed in any one of claims 1 to 7 when running the program instructions.

10. A storage medium storing program instructions, characterized in that, The program instructions, when executed, perform the document information extraction method based on image-text modal fusion as claimed in any one of claims 1 to 7.

Images