Document information extraction method and device based on image-text modal fusion and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a document information extraction method and device based on image-text modal fusion and a storage medium, and the method comprises the steps: obtaining a text block in a to-be-processed picture; extracting semantic vectors from the text blocks according to a pre-trained NLP model, extracting visual vectors by adopting a computer vision technology, and fusing the semantic vectors and the visual vectors through an attention mechanism to form feature representation of image-text fusion; according to the image-text fusion features, constructing a full-connection directed graph; on the constructed graph network, graph attention convolution of a'node-edge-node 'ternary feature group is executed; important'text-position-text 'ternary information is dynamically weighted through a self-attention mechanism; and outputting a feature vector fused with text, vision and position information. Through deep fusion of the text, vision and position information, the accuracy and robustness of text information matching are remarkably improved, and the method can adapt to a complex and changeable image text environment.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to image recognition and natural language processing technology of artificial intelligence, such as a document information extraction method, device and storage medium based on image and text modality fusion. Background Art

[0002] In the field of document intelligence, how to correctly extract structured information from unstructured text has become a hot topic, especially when dealing with the filling and verification of certificate information. Traditional template matching-based methods perform well in certificates with a single layout. However, when faced with license certificates with diverse layouts and long durations, such as business licenses, template matching encounters difficulties due to differences between different regions. After in-depth research on image characteristics, it is found that in the information matching process, in addition to the text content itself, the position of the text in the image and the visual features of the text also have a significant impact on the matching results.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0005] The embodiments of the present disclosure provide a document information extraction method, device and storage medium based on the fusion of graphic and text modalities. The method focuses on the ternary feature mining of "text-location-text" and aims to improve the distinction between semantically similar texts, thereby more effectively extracting structured information from complex unstructured texts.

[0006] In some embodiments, the method comprises:

[0007] Data processing: Get the text block in the image to be processed. The text block contains its visual features and text content in the image. The visual features include coordinate information.

[0008] Feature extraction and fusion: Semantic vectors are extracted from text blocks based on the pre-trained NLP model, and visual vectors are extracted from text blocks using computer vision technology. The semantic vectors and visual vectors are then fused through the attention mechanism to form a feature representation of image-text fusion.

[0009] Graph network construction: Based on the image and text fusion features, a fully connected directed graph is constructed;

[0010] Graph Attention Convolution: On the constructed graph network, perform graph attention convolution of the "node-edge-node" triple feature group;

[0011] Network feature weighting: Dynamically weight important "text-position-text" ternary information through the self-attention mechanism to enhance the expression of key information;

[0012] Output: After graph attention convolution and network feature weighting, the output is a feature vector that combines text, visual, and location information.

[0013] Optionally, in the data processing stage, obtaining the text block in the image to be processed includes:

[0014] Use the OCR system to process the input image and extract the text blocks in the image;

[0015] Get the detailed coordinates, size and text content of each text block.

[0016] Optionally, in the feature extraction and fusion stage, a self-attention layer is constructed, which accepts the semantic vector and the visual vector as input, and determines the correlation between the semantic vector and the visual vector by calculating the scaled dot product between the vectors.

[0017] Optionally, a multi-head self-attention mechanism is used to fuse the semantic vector and the visual vector to dynamically focus on key information in the text and image; based on the calculated correlation, the semantic vector and the visual vector are weighted and adjusted to achieve fusion.

[0018] Optionally, in the graph network construction stage, the semantic vector and the visual vector are fused as the initial features of the nodes in the graph network, and the position embedding vector of the current text and the adjacent text is calculated as the initial feature of the edge in the graph network.

[0019] Optionally, position embedding vectors are generated based on position indices through a set of predefined sine and cosine functions to capture the positional relationship between text blocks; the position embedding vectors are combined with word embedding or character embedding vectors to form a complete embedding representation of the model input.

[0020] Optionally, a GAT network is used to construct a graph structure, where each node represents a text instance, which contains fused semantic and visual feature vectors. The graph convolutional model updates the feature representation of the node by aggregating and transforming the node's neighbor information, and performs convolution on the "node-edge-node" ternary feature group, thereby capturing local and global structural information in the graph to better understand and process graph data.

[0021] In some embodiments, the apparatus comprises:

[0022] A data processing module is configured to obtain a text block in a to-be-processed image, wherein the text block includes its visual features and text content in the image, and the visual features include coordinate information;

[0023] The feature extraction and fusion module is configured to extract semantic vectors from text blocks according to the pre-trained NLP model, extract visual vectors from text blocks using computer vision technology, and then fuse the semantic vectors and visual vectors through the attention mechanism to form a feature representation of image-text fusion;

[0024] The graph network construction module is configured to construct a fully connected directed graph based on the graph-text fusion features;

[0025] A graph attention convolution module is configured to perform graph attention convolution of “node-edge-node” triple feature groups on the constructed graph network;

[0026] The network feature weighting module is configured to dynamically weight important "text-position-text" triple information through the self-attention mechanism to enhance the expression of key information;

[0027] The output module is configured to output a feature vector that combines text, visual and location information after graph attention convolution and network feature weighting processing.

[0028] In some embodiments, the device includes a processor and a memory storing program instructions, and the processor is configured to execute the aforementioned document information extraction method based on graphic and text modality fusion when running the program instructions.

[0029] In some embodiments, the storage medium stores program instructions, and when the program instructions are run, the aforementioned document information extraction method based on graphic and text modality fusion is executed.

[0030] The document information extraction method, device and storage medium based on image-text modality fusion provided by the embodiments of the present disclosure can achieve the following technical effects:

[0031] This disclosure combines graph network representation and graph attention convolution mechanism, so that the text information matching technology proposed in this patent has the following significant advantages:

[0032] Information richness: It integrates the text content, visual features, and positional relationships of text blocks to provide comprehensive semantic information, which helps to accurately distinguish texts with similar semantics.

[0033] Robustness: The graph attention network can adapt to text matching tasks in various complex scenarios and has good robustness to interference factors such as noise and deformation.

[0034] Efficiency: The structural design of the fully connected directed graph enables the network to quickly capture the global relationship between text blocks and improve the efficiency of text matching.

[0035] The present invention significantly improves the accuracy and robustness of text information matching by deeply integrating text, vision and location information. Compared with traditional methods, the present invention is more adaptable to complex and changing image and text environments, bringing substantial technological progress to the fields of information retrieval, intelligent question answering, etc.

[0036] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0038] Figure 1 It is a schematic diagram of a document information extraction principle based on image-text modality fusion provided by an embodiment of the present disclosure;

[0039] Figure 2 is a schematic diagram of a document information extraction device based on image-text modality fusion provided by an embodiment of the present disclosure;

[0040] Figure 3 is a schematic diagram of another document information extraction device based on image-text modality fusion provided by an embodiment of the present disclosure; DETAILED DESCRIPTION

[0041] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0042] The terms "first", "second", etc. in the embodiments of the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0043] Unless otherwise stated, the term "plurality" means two or more.

[0044] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0045] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0046] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0047] Information extraction is the process of extracting structured information from unstructured text. Traditional information extraction focuses on extracting entity and relationship information from plain text, but there is little research on visual text. Visually rich text data refers to data whose semantic structure is not only determined by the content of the text, but also contains rich semantic information in visual features such as text position relationship and font size. Certificates, receipts, invoices, etc. are all common visually rich text data, and most of the information is arranged in the form of key-value pairs. Through pre-training, these position information that is naturally aligned with the text can provide richer semantic information for downstream information extraction tasks.

[0048] Attention Mechanism: The attention mechanism is a brain signal processing mechanism unique to human vision. Human vision can quickly scan the global image to obtain the target area that needs to be focused on, which is generally referred to as the focus of attention, and then invest more attention resources in this area to obtain more detailed information about the target that needs to be focused on, while suppressing other useless information. This mechanism can greatly improve the efficiency and accuracy of visual information processing. In the field of deep learning, the attention mechanism is introduced to help the model automatically learn and understand the importance of data so that more computing resources can be allocated to important parts.

[0049] Self-Attention Mechanism: Self-attention is a special form of attention mechanism that allows the model to focus on different parts of a sequence when processing it. In self-attention, each input element interacts with all other elements in the sequence and a weighted representation is calculated based on these interactions. This mechanism is particularly suitable for processing sequence data, such as text data in natural language processing tasks. Through self-attention, the model can understand the dependencies between words in the sequence, regardless of their position in the sequence.

[0050] In short, the attention mechanism is a broader concept that can be applied to various scenarios, including but not limited to sequence-to-sequence tasks. The self-attention mechanism is a special application of the attention mechanism on sequence data (such as text), which enables the model to focus on the relationship between elements within the sequence. Therefore, although the self-attention mechanism is a type of attention mechanism, they differ in their specific implementation and application.

[0051] This application is based on the graph attention neural network technology of image-text modality fusion to solve the problem of text information matching, especially text matching in card images. This technology aims to improve the accuracy and discrimination of text matching by deeply fusing the text content, visual features and positional relationship between text blocks in the image with rich semantic information.

[0052] Combination Figure 1 The present disclosure provides a document information extraction device based on image-text modality fusion, comprising:

[0053] 1. Data processing module:

[0054] The input image is scanned using OCR (Optical Character Recognition) technology to accurately extract text blocks.

[0055] Get the detailed coordinates, size and text content of each text block.

[0056] 2. Feature extraction and fusion module:

[0057] Use pre-trained NLP (Natural Language Processing) models to convert text content into semantically rich semantic vectors.

[0058] Computer vision technology is used to extract visual features from text blocks and convert them into visual vectors.

[0059] The semantic vector and visual vector are fused through the attention mechanism to form a feature representation of image and text fusion.

[0060] 3. Graph network building module:

[0061] A fully connected directed graph is constructed using the image-text fusion features as node features.

[0062] The edge features between nodes are defined by the position embedding vectors between text blocks, which are generated by predefined sine and cosine functions to capture the positional relationship between texts.

[0063] 4. Graph Attention Convolution Module:

[0064] On the constructed graph network, graph attention convolution of the “node-edge-node” triple feature group is performed.

[0065] 5. Network feature weighting module:

[0066] The self-attention mechanism is introduced to dynamically weight the important "text-position-text" ternary information and enhance the expression of key information.

[0067] 6. Output module:

[0068] After being processed by the graph attention network, the output is a feature vector that combines text, visual and location information.

[0069] Among them, in the text information matching task, high-precision text information matching is achieved by calculating the similarity between these feature vectors.

[0070] It can be understood that fusing semantic vectors and visual vectors based on the self-attention mechanism is an effective method, which allows the model to dynamically focus on key information in different modalities (text and image) during the fusion process. First, the text information in the image is extracted through OCR technology, which is used to convert the text in the image into editable and searchable text.

[0071] Since OCR itself does not provide semantic vectors for text, it is necessary to use a pre-trained NLP model to convert text into a vector representation rich in semantic information. At the same time, computer vision technology is used to extract features from the image within the text coordinates and convert them into corresponding visual vectors.

[0072] Next, a self-attention layer is constructed, which accepts the semantic vector and the visual vector as input and determines the degree of association between them by calculating the scaled dot product between the vectors. This mechanism not only considers the intra-modality relationship (i.e. the context within the text or within the image), but also the inter-modality relationship (i.e. the interaction between text and image).

[0073] By adopting a multi-head self-attention design, different heads can focus on different modalities or modal combinations. Finally, the semantic vector and the visual vector are weighted and adjusted according to the calculated correlation, and fused through weighted summation or weighted concatenation. In this process, the fused vector not only integrates information from different modalities, but also balances them according to the weights obtained in the self-attention mechanism, thereby ensuring that the model can dynamically focus on the key information in the text and image when making decisions.

[0074] Among them, position embedding uses a set of predefined sine and cosine functions to generate position embeddings based on the index of each position. The embedding of each position is calculated by points on sine and cosine waves of different frequencies. A key feature of this method is that it does not depend on the model training process, the embedding is fixed and the same for all input sequences. Through position embedding, the model is able to capture the positional relationship between words or characters, which is crucial for understanding the structure and semantics of sentences.

[0075] Position embedding can help the model distinguish this relationship and thus understand the meaning of the sentence more accurately. Position information is mapped to a continuous vector space to generate position embedding vectors. These vectors are combined with word embedding or character embedding vectors to form a complete embedding representation of the model input.

[0076] The GAT network is used to construct a graph structure, where each node represents a text instance, and its feature is the fused semantic and visual feature vector. The initial features of the edge are defined by the position embedding vector. The graph convolution model updates the feature representation of the node by aggregating and transforming the node's neighbor information, and performs convolution on the "node-edge-node" triple feature group. This process helps the model capture local and global structural information in the graph, thereby better understanding and processing graph data.

[0077] As an example, the present disclosure provides a document information extraction method based on image-text modality fusion as follows:

[0078] (1) Data processing

[0079] Each input image is processed using an OCR (Optical Character Recognition) system to extract text blocks in the image. Each text block contains its coordinate information in the image (such as the coordinates of the upper left corner and the lower right corner), visual features such as length and width, and text content.

[0080] (2) Feature Fusion

[0081] According to the OCR results, the semantic vector is extracted by NLP model, and the visual vector is extracted by visual computer technology. The attention mechanism is introduced to fuse the two vectors as the initial features of the node. The position embedding vector of the current text and the adjacent text is calculated as the initial feature of the edge.

[0082] (3) Graph Network Construction

[0083] The multimodal feature fusion of images and texts is used as the initial feature of nodes, and the position embedding vector is used as the initial feature of edges to construct a graph network and form a fully connected directed graph.

[0084] (4) Convolution of ternary feature groups

[0085] On the constructed fully connected directed graph, a graph attention convolution operation is performed on the "node-edge-node" ternary feature group. The convolution operation can comprehensively consider the node features, edge features, and features of adjacent nodes, thereby extracting richer fusion features.

[0086] (5) Network feature weighting

[0087] The graph network introduces a self-attention mechanism, which calculates the attention weight of each triple and weighted aggregate features, so that the network can focus on the more important "text-position-text" triple information.

[0088] (6) Feature fusion and output

[0089] After the convolution and self-attention mechanism of the graph attention network, each node will get a feature vector that combines text content, visual features and positional relationships. These feature vectors contain both the semantic information of the text block and the spatial relationship between the text blocks.

[0090] In the text information matching task, the similarity between the feature vectors of text blocks in different images is calculated to determine the matching degree of the text information.

[0091] The present disclosure proposes a document information extraction method and device based on image-text modality fusion to solve the problem of text information matching. After each image passes through the OCR system, a set of text blocks will be obtained, and each text block contains its coordinate information and text content in the image. The graph attention network is used to model the text content and visual features in the text block and the positional relationship between the texts. The fusion features enrich the semantic information of card images and further improve the discrimination of semantically similar texts.

[0092] This work uses the text blocks obtained by OCR to form a fully connected directed graph. Each text block constitutes a node, and each node is connected to all other nodes. The initial features of the node are composed of the text content and visual features of the text block. The initial features of the edge are composed of the coordinates, length, width and other information of the current text block, as well as the positional relationship with the adjacent text blocks.

[0093] Secondly, the network performs convolution on the "node-edge-node" ternary feature group. The self-attention mechanism allows the network to select more noteworthy information from all the directional triplets in the fully connected directed graph and weighted aggregate features. It pays more attention to the correlation of the ternary information of "text-position-text" and effectively distinguishes texts.

[0094] The document information extraction technology proposed in this disclosure has the following significant advantages:

[0095] Information richness: It integrates the text content, visual features, and positional relationships of text blocks to provide comprehensive semantic information, which helps to accurately distinguish texts with similar semantics.

[0096] Robustness: The graph attention network can adapt to text matching tasks in various complex scenarios and has good robustness to interference factors such as noise and deformation.

[0097] Efficiency: The structural design of the fully connected directed graph enables the network to quickly capture the global relationship between text blocks and improve the efficiency of text matching.

[0098] The present invention significantly improves the accuracy and robustness of text information matching by deeply integrating text, vision and location information. Compared with traditional methods, the present invention is more adaptable to complex and changing image and text environments, bringing substantial technological progress to the fields of information retrieval, intelligent question answering, etc.

[0099] Combination Figure 2 As shown, the embodiment of the present disclosure provides a document information extraction device 200 based on image-text modality fusion, including a data processing module 201, a feature extraction and fusion module 202, a graph network construction module 203, a graph attention convolution module 204, a network feature weighting module 205 and an output module 206.

[0100] The data processing module 201 is configured to obtain a text block in a to-be-processed image, wherein the text block includes its visual features and text content in the image, and the visual features include coordinate information;

[0101] The feature extraction and fusion module 202 is configured to extract semantic vectors from the text block according to the pre-trained NLP model, extract visual vectors from the text block using computer vision technology, and then fuse the semantic vectors and the visual vectors through an attention mechanism to form a feature representation of image-text fusion;

[0102] A graph network construction module 203 is configured to construct a fully connected directed graph according to the graph-text fusion features;

[0103] A graph attention convolution module 204 is configured to perform graph attention convolution of a “node-edge-node” triple feature group on the constructed graph network;

[0104] The network feature weighting module 205 is configured to dynamically weight important “text-position-text” ternary information through a self-attention mechanism to enhance the expression of key information;

[0105] The output module 206 is configured to output a feature vector that integrates text, vision and location information after graph attention convolution and network feature weighting processing.

[0106] Combination Figure 3As shown, the embodiment of the present disclosure provides a document information extraction device 300 based on image-text modality fusion, including a processor (processor) 304 and a memory (memory) 301. Optionally, the device may also include a communication interface (Communication Interface) 302 and a bus 303. Among them, the processor 304, the communication interface 302, and the memory 301 can communicate with each other through the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can call the logic instructions in the memory 301 to execute the method for ... of the above embodiment.

[0107] In addition, the logic instructions in the memory 301 described above can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0108] The memory 301 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 304 executes the function application and data processing by running the program instructions / modules stored in the memory 301, that is, the method for document information extraction based on image-text modality fusion in the above embodiment is implemented.

[0109] The memory 301 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 301 may include a high-speed random access memory and may also include a non-volatile memory.

[0110] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the document information extraction method based on the fusion of graphic and text modalities.

[0111] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0112] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0113] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the scope of protection. As used in the description in the text, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0114] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians may clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0115] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0116] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A document information extraction method based on image-text modality fusion, characterized in that: include: Data processing: obtaining a text block in the image to be processed, wherein the text block includes its visual features and text content in the image, and the visual features include coordinate information; Feature extraction and fusion: Semantic vectors are extracted from text blocks based on the pre-trained NLP model, and visual vectors are extracted from text blocks using computer vision technology. The semantic vectors and visual vectors are then fused through the attention mechanism to form a feature representation of image-text fusion. Graph network construction: Based on the image and text fusion features, a fully connected directed graph is constructed; Graph Attention Convolution: On the constructed graph network, perform graph attention convolution of the "node-edge-node" triple feature group; Network feature weighting: Dynamically weight important "text-position-text" ternary information through the self-attention mechanism to enhance the expression of key information; Output: After graph attention convolution and network feature weighting, the output is a feature vector that combines text, visual, and location information.

2. The method according to claim 1, characterized in that In the data processing stage, obtaining the text block in the image to be processed includes: Use the OCR system to process the input image and extract the text blocks in the image; Get the detailed coordinates, size and text content of each text block.

3. The method according to claim 1, characterized in that In the feature extraction and fusion stage, A self-attention layer is constructed, which accepts the semantic vector and the visual vector as input and determines the correlation between the semantic vector and the visual vector by calculating the scaled dot product between the vectors.

4. The method according to claim 3, characterized in that A multi-head self-attention mechanism is used to fuse semantic vectors and visual vectors to dynamically focus on key information in text and images. According to the calculated correlation, the semantic vector and the visual vector are weighted and adjusted to achieve fusion.

5. The method according to any one of claims 1 to 4, characterized in that: In the graph network construction phase, The semantic vector and visual vector are fused as the initial features of the nodes in the graph network, and the position embedding vector of the current text and the adjacent text is calculated as the initial feature of the edge in the graph network.

6. The method according to claim 5, characterized in that The position embedding vector is generated according to the position index through a set of predefined sine and cosine functions to capture the position relationship between text blocks; The position embedding vector is combined with the word embedding or character embedding vector to form a complete embedding representation of the model input.

7. The method according to claim 5, characterized in that A graph structure is constructed using a GAT network, where each node represents a text instance, and the text instance includes fused semantic and visual feature vectors; The graph convolutional model updates the feature representation of nodes by aggregating and transforming the node's neighbor information, and performs convolution on the "node-edge-node" triple feature group to capture local and global structural information in the graph to better understand and process graph data.

8. A document information extraction device based on image-text modality fusion, characterized in that: include: A data processing module is configured to obtain a text block in a to-be-processed image, wherein the text block includes its visual features and text content in the image, and the visual features include coordinate information; The feature extraction and fusion module is configured to extract semantic vectors from text blocks according to the pre-trained NLP model, extract visual vectors from text blocks using computer vision technology, and then fuse the semantic vectors and visual vectors through the attention mechanism to form a feature representation of image-text fusion; The graph network construction module is configured to construct a fully connected directed graph based on the graph-text fusion features; The graph attention convolution module is configured to perform graph attention convolution of "node-edge-node" triple feature groups on the constructed graph network; The network feature weighting module is configured to dynamically weight important "text-position-text" triple information through the self-attention mechanism to enhance the expression of key information; The output module is configured to output a feature vector that combines text, visual and location information after graph attention convolution and network feature weighting processing.

9. A document information extraction device based on image-text modality fusion, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the document information extraction method based on image-text modality fusion as described in any one of claims 1 to 7 when running the program instructions.

10. A storage medium storing program instructions, characterized in that: When the program instructions are run, they execute the document information extraction method based on image-text modality fusion as described in any one of claims 1 to 7.

Images