Generative visual document zero sample information extraction method based on large model

Abstract

The invention provides a large model-based generative visual document zero sample information extraction method. The method comprises six unit architectures: data preprocessing, a multi-modal feature extraction task, a layout enhancement task based on a large language model, a feature fusion task, an adapter pre-training task and an information extraction task based on the large language model. According to the method, feature fusion and processing are carried out, so that a large language model executes an information extraction task; training is carried out through a text region prediction task and a coordinate mapping text recovery task so as to enhance the feature processing capability; according to the method, all aspects of features of the visual document and the understanding ability of the large language model are fully utilized, the accuracy of document information extraction is improved, document information extraction of predefined entity categories is not needed, therefore, the service range of a supply chain semantic dictionary is rapidly expanded, the flexibility and efficiency of information processing are remarkably improved, and the method is suitable for application and popularization.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and computer vision technology, and in particular to a large model-based generative visual document zero-sample information extraction method. Background Art

[0002] Traditional information extraction technology mainly focuses on extracting entity and relationship information from plain text, while there are relatively few studies on text containing rich visual information. In practical applications, especially in industrial supply chain scenarios, visual documents are the mainstream data model. A large amount of data is stored in unstructured forms such as pictures, PDF files, and paper documents, including: supply chain procurement contracts, work orders, receipts, invoices, purchase orders, test reports, supplier financial statements, etc. These documents are full of rich visual information and have important business value. Therefore, there is an urgent need to develop technologies that can automatically extract and structure key information in these documents.

[0003] Different from traditional text-based information extraction tasks, visual document information extraction tasks rely on both textual and visual information; therefore, in information extraction tasks for visual documents, how to make full use of textual, visual and layout information is crucial.

[0004] Traditional rule-based and machine learning-based visual document information extraction models require a lot of manpower, and the model's capabilities and versatility are also insufficient. With the development of deep learning technology, most existing visual document information extraction models are implemented using the following technical routes:

[0005] First, the visual document is scanned using OCR technology to obtain text and layout information;

[0006] Secondly, the text, layout and visual image information are organized and analyzed using large-scale pre-trained deep neural networks;

[0007] Although models such as LayoutLM have been produced that combine 2D position information, image embedding and text for pre-training and have achieved good results to a certain extent, the "pre-training-fine-tuning" model still faces many shortcomings in practical applications.

[0008] In practical applications, due to the diversity of information formats and types, and the existence of a large number of unlabeled entity types, it is necessary to iteratively expand the domain of new knowledge that is not included in the semantic dictionary business, and continuously incorporate supply chain behaviors to discover new relationship links; most of the existing visual document information extraction methods can only be trained on labeled data sets, with weak generalization capabilities, and it is difficult to cope with the recognition of undefined entity categories, especially when faced with new style documents; in actual scenarios, the number of entity categories is huge and often expands, and existing methods are difficult to meet the needs of quickly enriching and iterating semantic dictionary services for new scenarios in the supply chain.

[0009] In addition, current methods mostly rely on sequence labeling models, which are limited by the accuracy of OCR tools and the disorder of text. A lot of effort is needed to perform fine-grained labeling of documents, which further affects the generalization of the model. The rise of large language models (LLMs) provides new ideas for solving these problems. Large language models have demonstrated excellent capabilities in text understanding and generation, can generalize across multiple fields and tasks, and have the potential to achieve effective extraction of unlabeled entity information in documents. However, how to effectively apply large language models to information extraction from visual documents, especially in the absence of targeted training, how to extract fine-grained visual features, is still an important issue to be solved. Summary of the invention

[0010] In order to solve the above technical problems, the present invention provides a generative visual document zero-shot information extraction method based on a large model, which solves the problem of zero-shot information extraction of visual documents and extracts unlabeled entity information in documents in the absence of targeted training to meet the needs of rapidly enriching and iterating semantic dictionary services for new scenarios in the supply chain. At the same time, the document-level multimodal features are aligned with the text layout features to enhance the model's understanding and processing capabilities.

[0011] A generative visual document zero-shot information extraction method based on a large model, including six unit architectures: data preprocessing, multimodal feature extraction tasks, layout enhancement tasks based on a large language model, feature fusion tasks, adapter pre-training tasks, and information extraction tasks based on a large language model;

[0012] The data preprocessing is responsible for scanning document images and extracting text content and layout information;

[0013] As an example, the data preprocessing utilizes the optical character recognition (OCR) method to scan the document image and extract the text content of the document image and its corresponding layout information, thereby providing a modal information basis for the multimodal feature extraction task of the image.

[0014] The multimodal feature extraction task is responsible for integrating the text content, layout information and visual information; extracting these three sets of information features through the document pre-training model, combining and encoding them, and outputting multimodal features Fc, thereby enhancing the understanding of the document content by the large language model and overcoming the limitations of single modality feature extraction;

[0015] As an example, the document image is taken as a kind of the visual information.

[0016] As an example, the multimodal feature extraction task uses a document pre-trained model to integrate the three sets of information to fully exploit the natural alignment of images and text,layouts in visual documents.

[0017] The layout enhancement task based on the large language model is responsible for analyzing the size and position information of the text block, outputting the encoded layout enhancement feature FL, and improving the understanding ability of the large language model in the visual document information extraction task;

[0018] As an example, the text content and layout information are further interpreted and enhanced by inputting the text content and layout information into a large language model with fixed parameters, analyzing the size of the text block, the global position of the text block in the document, and the relative positions between text blocks.

[0019] The feature fusion task is responsible for building an adapter to bridge and convert the multimodal feature Fc and the layout enhancement feature FL to achieve effective fusion of the two, and adjust the dimensions of the multimodal feature Fc and the layout enhancement feature FL to match the large language model requirements of the information extraction task stage, and improve the flexibility and efficiency of processing multimodal data; finally, the hybrid feature H is output F .

[0020] The adapter pre-training task is responsible for training the adapter to enhance its feature fusion capability through text region prediction and coordinate mapping text recovery tasks.

[0021] As an example, the present invention freezes the rest of the entire network and uses the data set to train the adapter.

[0022] As an example, the data set includes a large number of document images, text contents in the document images, and corresponding layout information.

[0023] The information extraction task based on the large language model is responsible for converting the mixed feature H F The large language model is input with prompt instructions, and the understanding ability of the large language model is used to generate structured text to achieve efficient extraction of document information.

[0024] As an example, the mixed features after the feature fusion task are input into the large language model, and the designed prompt instructions are input at the same time. The understanding ability of the large language model is used to generate structured text and realize the document information extraction task.

[0025] As an example, the hybrid feature H F It is: the feature obtained by merging the multimodal feature Fc and the layout enhancement feature FL.

[0026] Beneficial effects of the present invention:

[0027] The present invention extracts text, vision, layout and other features from visual document images, and fuses and processes the features for a large language model to perform information extraction tasks. In the adapter pre-training stage, training is performed through text region prediction tasks and coordinate mapping text recovery tasks to enhance feature processing capabilities. Finally, information extraction tasks based on a large language model are performed to make full use of various features of visual documents and the understanding capabilities of the large language model to improve the accuracy of document information extraction.

[0028] The present invention is used to realize document information extraction without pre-defining entity categories, thereby quickly expanding the business scope of the supply chain semantic dictionary and significantly improving the flexibility and efficiency of information processing; compared with previous methods, the method proposed in the present invention converts the visual document information extraction task into a sequence-to-sequence generative task, fully utilizing the text understanding and generation capabilities of the large model, and has higher flexibility and efficiency.

[0029] In the field of academic research, especially in the advanced applications of cross-modal information processing and iterative domain expansion technology, the present invention has deepened the understanding of cross-modal learning theory and explored the dynamic update and expansion of automated knowledge graphs, which has significant advantages; it can not only analyze known supply chain data, but also has the ability to continuously incorporate new information and knowledge and continuously expand the supply chain knowledge graph, so as to cope with dynamically changing business needs, and has broad application prospects in fields such as supply chain management.

[0030] The invention has scientific and reliable design, is easy to operate, has simple configuration requirements, and is suitable for popularization and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the overall process of a large-model-based generative visual document zero-sample information extraction method of the present invention.

[0032] Figure 2 Schematic diagram of the overall module architecture of a large-model-based generative visual document zero-sample information extraction method of the present invention. DETAILED DESCRIPTION

[0033] Below, reference Figure 1 to Figure 2 As shown, a generative visual document zero-shot information extraction method based on a large model includes six unit architectures: data preprocessing, multimodal feature extraction task, layout enhancement task based on a large language model, feature fusion task, adapter pre-training task and information extraction task based on a large language model;

[0034] The data preprocessing is responsible for scanning document images and extracting text content and layout information;

[0035] As an example, the data preprocessing utilizes the optical character recognition (OCR) method to scan the document image and extract the text content of the document image and its corresponding layout information, thereby providing a modal information basis for the multimodal feature extraction task of the image.

[0036] The multimodal feature extraction task is responsible for integrating the text content, layout information and visual information; extracting these three sets of information features through the document pre-training model, combining and encoding them, and outputting multimodal features Fc, thereby enhancing the understanding of the document content by the large language model and overcoming the limitations of single modality feature extraction;

[0037] As an example, the document image is taken as a kind of the visual information.

[0038] As an example, the multimodal feature extraction task uses a document pre-trained model to integrate the three sets of information to fully exploit the natural alignment of images and text,layouts in visual documents.

[0039] The layout enhancement task based on the large language model is responsible for analyzing the size and position information of the text block, outputting the encoded layout enhancement feature FL, and improving the understanding ability of the large language model in the visual document information extraction task;

[0040] As an example, the text content and layout information are further interpreted and enhanced by inputting the text content and layout information into a large language model with fixed parameters, analyzing the size of the text block, the global position of the text block in the document, and the relative positions between text blocks.

[0041] The feature fusion task is responsible for building an adapter to bridge and convert the multimodal feature Fc and the layout enhancement feature FL to achieve effective fusion of the two, and adjust the dimensions of the multimodal feature Fc and the layout enhancement feature FL to match the large language model requirements of the information extraction task stage, and improve the flexibility and efficiency of processing multimodal data; finally, the hybrid feature H is output F .

[0042] The adapter pre-training task is responsible for training the adapter to enhance its feature fusion capability through text region prediction and coordinate mapping text recovery tasks.

[0043] As an example, the present invention freezes the rest of the entire network and uses the data set to train the adapter.

[0044] As an example, the data set includes a large number of document images, text contents in the document images, and corresponding layout information.

[0045] The information extraction task based on the large language model is responsible for converting the mixed feature H F The large language model is input with prompt instructions, and the understanding ability of the large language model is used to generate structured text to achieve efficient extraction of document information.

[0046] As an example, the mixed features after the feature fusion task are input into the large language model, and the designed prompt instructions are input at the same time. The understanding ability of the large language model is used to generate structured text and realize the document information extraction task.

[0047] As an example, the hybrid feature H F It is: the feature obtained by merging the multimodal feature Fc and the layout enhancement feature FL.

[0048] In order to better illustrate the design principle of the present invention, a specific application example is given as follows:

[0049] A large model-based generative visual document zero-shot information extraction method, comprising:

[0050] S1. Data preprocessing:

[0051] Use the OCR tool (Paddle OCR) to convert the document image into text content Text and layout information Box;

[0052] The text content is in the form of plain text, and the text layout information is used to describe the area where the text is located in the image. The text area recognized by OCR is represented as a rectangular box, and the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of these rectangles define the text layout information Box: Box = [x1, y1, x2, y2];

[0053] S2, multimodal feature extraction task:

[0054] The image of a visual document can be used as a visual information Image, which has a natural alignment with its text content Text and layout information Box;

[0055] It is difficult to fully utilize the connection between the modalities by extracting visual features or text content features alone. Therefore, the document pre-trained model LayoutLMv3 is used as the M-Encoder to combine and align the text content, layout information and visual information, and comprehensively extract the multimodal features Fc of the document:

[0056] Fc=M-Encoder(Texy, Box, Image);

[0057] S3. Layout enhancement task based on large language model:

[0058] Layout features play a key role in visual documents. The layout information extracted by OCR makes it difficult for the large language model to effectively understand the layout of the document in the information extraction task, which in turn limits the large language model's zero-sample document understanding ability. Therefore, the large language model is used to further interpret and enhance the layout information through context learning. The large language model that performs this task is called "Layout-augment Large Model (LAM, Layout-augment LLM)". LAM receives text content Text and layout information Box, and analyzes the size of the text block, the global position of the text block in the document, and the relative position between the text block and other text blocks through instruction-guided context learning, and encodes the above information to obtain the layout enhancement feature FL:

[0059] F L =LAM(Text, Box);

[0060] The instruction template for context learning is as follows:

[0061]

[0062] S4, feature fusion task:

[0063] Design a specific network layer as an adapter to bridge the features of various modalities output from S2 and S3 and convert them to meet the requirements of the large language model in the information extraction task stage;

[0064] The adapter consists of three fully connected layers, each followed by a ReLU activation function. The initial layer transforms the input features into a hidden representation; this hidden representation is further refined by a second layer of the same size, which enhances the features.

[0065] Finally, the third layer projects the refined features to the input dimension D required by the large language model in the information extraction task stage LLM The final output is the mixed feature H F .H FIt will be input into the pre-trained large language model for subsequent task-specific training and information extraction. F The calculation formula is as follows:

[0066] H F =Concat(Adapter(F C , D LLM ), F L )

[0067] Among them, Concat means concatenating the two parts in dimension.

[0068] S5: Adapter pre-training:

[0069] Freeze the rest of the network and train the adapter using two different tasks to enhance its ability to fuse features and understand document layout. The tasks are: (a) Text region prediction: The goal of this task is to predict the bounding box of a specific text region given the text content. (b) Coordinate mapping text restoration: In this task, the goal is to restore the text content within the bounding box of a given text region. The training loss uses the cross entropy loss:

[0070]

[0071] Among them, y i is the true label, LLM(H F ) is the mixed feature H F The probability of the predicted label output after inputting the large language model in the information extraction task stage.

[0072] S6, information extraction task based on large language model: Use the frozen large language model (LLM) to receive the mixed feature H output by S4 F And the instructions for information extraction, based on the understanding ability of the large language model, generate the structured information corresponding to the document. Among them, the instructions for information extraction are:

[0073]

[0074] in, <H F > is the mixed features output by the pre-trained adapter.

[0075] The above are only preferred embodiments of the present invention. It should be understood that the description of the above embodiments is only used to help understand the method and core ideas of the present invention, and is not used to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, etc. made within the ideas and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A generative visual document zero-shot information extraction method based on a large model, characterized in that: It includes six unit architectures: data preprocessing, multimodal feature extraction task, layout enhancement task based on large language model, feature fusion task, adapter pre-training task and information extraction task based on large language model; The data preprocessing is responsible for scanning document images and extracting text content and layout information; The multimodal feature extraction task is responsible for integrating the text content, layout information and visual information; extracting these three sets of information features through the document pre-training model, combining and encoding them, and outputting multimodal features Fc, thereby enhancing the understanding of the document content by the large language model and overcoming the limitations of single modality feature extraction; The layout enhancement task based on the large language model is responsible for analyzing the size and position information of the text block, outputting the encoded layout enhancement feature FL, and improving the understanding ability of the large language model in the visual document information extraction task; The feature fusion task is responsible for building an adapter to bridge and convert the multimodal feature Fc and the layout enhancement feature FL to achieve effective fusion of the two, and adjust the dimensions of the multimodal feature Fc and the layout enhancement feature FL to match the large language model requirements of the information extraction task stage, and improve the flexibility and efficiency of processing multimodal data; finally, the hybrid feature H is output F ; The adapter pre-training task is responsible for training the adapter to enhance its feature fusion capability through text region prediction and coordinate mapping text recovery tasks; The information extraction task based on the large language model is responsible for converting the mixed feature H F The large language model is input with prompt instructions, and the understanding ability of the large language model is used to generate structured text to achieve efficient extraction of document information.

2. According to claim 1, a large model-based generative visual document zero-shot information extraction method is characterized in that: The data preprocessing utilizes an optical character recognition method to scan the document image and extract the text content of the document image and its corresponding layout information, thereby providing a modal information basis for the multimodal feature extraction task of the image.

3. According to claim 1, a large model-based generative visual document zero-shot information extraction method is characterized in that: The multimodal feature extraction task uses a document pre-trained model to integrate the three sets of information to fully exploit the natural alignment properties of images and text,layouts in visual documents.

4. The method for extracting zero-shot information from a large-scale generative visual document according to claim 1, characterized in that: By inputting the text content and layout information into a large language model with fixed parameters, the size of the text block, the global position of the text block in the document, and the relative position between the text blocks are analyzed to further interpret and enhance the text content and layout information.

5. The method for extracting zero-shot information from a large-scale generative visual document according to claim 1, characterized in that: The adapter is trained with a data set; the data set includes a large number of document images, text contents in the document images, and corresponding layout information.

6. The method for extracting zero-shot information from a large-scale generative visual document according to claim 1, characterized in that: The mixed features after the feature fusion task are input into the large language model, and the designed prompt instructions are input at the same time. The understanding ability of the large language model is used to generate structured text to achieve the document information extraction task.

7. The method for extracting zero-shot information from a large-scale generative visual document according to claim 1, characterized in that: The mixed characteristic H F It is: the feature obtained by merging the multimodal feature Fc and the layout enhancement feature FL.

8. The method for extracting zero-shot information from a large-scale generative visual document according to claim 7, characterized in that: The multimodal feature Fc formula is designed as: Fc=M-Encoder(Text, Box, Image); Among them: Image is: visual information, Text is: text content, Box is: layout information, and M-Encoder is: document pre-training model LayoutLMv3.

9. The method for extracting zero-shot information from a generative visual document based on a large model according to claim 7, characterized in that: The layout enhancement feature FL formula is designed as: F L =LAM(Text,Box); Among them: LAM stands for layout-augment large model, Layout-augment LLM.

10. The method for extracting zero-shot information from a large-scale generative visual document according to claim 7, characterized in that: The mixed characteristic H F The formula is designed as: H F =Concat(Adapter(F c ,D LLM ),F L ); Concat means concatenating the multimodal feature Fc and the layout enhancement feature FL in dimension, and Adapter is: adapter, D LLM : The input dimension required for a large language model in the information extraction task stage.

Images