Document processing method of document visual question and answer task based on multi-granularity prediction

Abstract

The invention provides a document processing method for a document visual question and answer task based on multi-granularity prediction. The document processing method comprises the following steps: S1, identifying a format of an input document and extracting text and image information in the input document; s2, performing semantic understanding and context analysis on the text by adopting a natural language processing technology; s3, performing joint embedding and alignment on the text information and the visual information through a deep learning model, and capturing explicit semantic information in the document; and S4, based on the question input by the user, performing multi-granularity matching in combination with the text and the visual information, and generating a predicted answer. The method is characterized in that a multi-granularity retrieval and reasoning mechanism is introduced, and the accuracy of answer prediction is remarkably improved by enhancing the modeling capability of the model for the short-distance context relationship. Besides, the adopted verification editing reasoning framework improves the interpretability of the answer prediction process, so that the generated answers have higher transparency and credibility, and the method is particularly suitable for task environments with higher precision requirements.

Description

Technical Field

[0001] The present invention relates to document processing technology, and in particular to a document processing method for a document visual question answering task based on multi-granularity prediction, which is particularly suitable for processing documents containing multiple information elements such as images, texts, tables, etc. Background Art

[0002] In recent years, with the development of document images and multimodal technology, the document visual question answering (Document VQA) task has gradually attracted attention. The document visual question answering task not only needs to parse the visual content in the image, but also needs to process the structured text information in the image and the relationship between the image and the text. Such tasks usually involve multiple levels of information acquisition, including text, tables, charts, handwritten text and other forms of data in the document. How to efficiently integrate and reason in complex multimodal information has become a key problem that needs to be solved in this field.

[0003] Traditional document processing methods usually adopt a unified feature extraction method, but these methods often ignore the multi-granularity characteristics of document content. The information in the document is significantly hierarchical. Different granularity information such as chapters, paragraphs, tables and images in the document present different semantic characteristics. Directly treating this information equally may cause the model to fail to fully utilize the correlation between multi-level information, thus affecting the overall reasoning performance.

[0004] To solve this problem, a document visual question answering method based on multi-granularity prediction has emerged. This method extracts information of different granularities from the document, processes the information separately, extracts the respective semantic features layer by layer, and finally fuses them at a high level. In this way, it can not only enhance the sensitivity to details, but also improve the understanding of the overall structure and semantic relationship of the document, thereby achieving better performance in multimodal question answering tasks. Summary of the invention

[0005] The present invention proposes a document processing method for document visual question answering tasks based on multi-granularity prediction, which aims to improve the reasoning ability and accuracy of the document visual question answering system in complex documents through multi-level and multi-scale feature extraction and fusion. By modeling the document content in a hierarchical and multi-granular manner, the present invention can effectively improve the ability to solve document visual question answering tasks, especially when facing diverse document content and complex questions, it can significantly improve the accuracy of question answering, and has broad application prospects.

[0006] To solve the above problems, this application adopts the following technical solutions:

[0007] 1. A document processing method based on a document visual question answering task, the method comprising the following steps:

[0008] S1, identify the format of the input document and extract text and image information in the document;

[0009] S2, using natural language processing technology to perform semantic understanding and context analysis on text information in documents;

[0010] S3, jointly embed and align text information and visual information through deep learning models;

[0011] S4. Generate answers to questions based on the questions input by the user by combining text and visual information.

[0012] The beneficial effects achieved by the present invention are:

[0013] Improve the accuracy of answer prediction: The present invention ensures a comprehensive understanding of the document content through a multi-granularity hybrid retrieval mechanism, combining the global structural information and local details of the document, thereby improving the ability to locate the key position of the answer.

[0014] Enhance the interpretability and transparency of the model: The verification and editing reasoning framework introduced in this paper makes the answer prediction process not only highly accurate but also well interpretable. Through the iterative reasoning process, the model can gradually correct the initial predicted answer and clearly display the reasoning path and basis for answer generation.

[0015] Improved robustness and reliability of document processing: The multi-granularity retrieval and reasoning framework of the present invention can effectively cope with complex document content. Especially when processing structured data such as charts and tables, the system can perform fine-grained retrieval and reasoning based on the multi-level information of the document to ensure the reliability of the answer.

[0016] Optimized computing efficiency and processing speed: Compared with the traditional global attention-based model, the present invention reduces the consumption of computing resources and improves the processing speed of document question answering by combining multi-granularity retrieval and local attention mechanism.

[0017] Improved utilization efficiency of document structured information: Traditional document visual question answering models usually lack effective utilization of document structured information (such as charts, tables, etc.), while the present invention significantly improves the model's ability to process such structured information through a multi-granularity retrieval mechanism, enabling the model to more accurately extract key information when answering questions involving charts, lists, etc.

[0018] Strong scalability and adaptability: The framework of the present invention has strong scalability in different fields and application scenarios. By flexibly adjusting the retrieval and reasoning mechanism, it can be customized and optimized according to different document types (such as contracts, reports, technical documents, etc.) or field requirements to meet a wider range of document question and answer needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0020] Figure 1 It is a document processing flow chart.

[0021] Figure 2 It is a diagram showing the semantic information extraction model.

[0022] Figure 3 It is a diagram of the hierarchical answer prediction module based on multi-granularity matching.

[0023] Figure 4 It is a schematic diagram of the short-distance information interaction module. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, for step S1, identifying the format of the input document and extracting the text and image information in the document, it is necessary to use document format recognition technology, text extraction technology, image extraction technology, and data storage and formatting processing technology. The step S1 specifically includes:

[0026] S11. Format recognition: The file format of the input document is automatically detected through the format recognition module, and the specific type of the document (PDF, DOCX, HTML, TIFF, etc.) is determined using the file extension, file header information or metadata embedded in the document.

[0027] S12. Text extraction: According to the identified file format, the corresponding text extraction technology is used. For PDF format, the editable text is extracted by parsing the PDF structure and optical character recognition (OCR) technology; for Word format, the text content is obtained by calling the python-docx document parsing library; for other formats, the system will select the appropriate extraction method according to the format characteristics.

[0028] S13. Image extraction: After text extraction, the system will analyze the image data in the document. For PDF files, use the image extraction tool PyMuPDF to extract embedded images; for HTML or Word files, scan and extract embedded image files.

[0029] For step S2, natural language processing technology is used to perform semantic understanding and context analysis on the text information in the document. The step S2 specifically includes:

[0030] S21. Text preprocessing: Before semantic understanding and context analysis, the extracted text is first preprocessed. This process includes removing irrelevant information such as stop words, punctuation marks, special characters, etc.; performing word segmentation to split the text into independent vocabulary units; and performing word form restoration (such as restoring verbs to their root form) to standardize the text content and ensure the accuracy of subsequent analysis.

[0031] S22. Named Entity Recognition: Through named entity recognition technology, key entities in the text, such as names of people, places, organizations, etc., are automatically identified and classified. This step helps to extract key information from the document and provides a basis for further semantic understanding.

[0032] S23. Syntactic analysis and dependency analysis: Syntactic analysis and dependency analysis help the system clarify the grammatical functions and logical relationships between words by analyzing the sentence structure and the dependencies between words in the document. Syntactic analysis constructs a syntax tree to identify the hierarchical relationships between the components of a sentence (such as subject, predicate, object, etc.); dependency analysis reveals the dependencies between words and clarifies grammatical connections such as subject and predicate, verb and object, and modifiers. Through these analyses, the system can accurately understand the grammatical structure of the text and provide support for subsequent semantic understanding and information extraction.

[0033] S24, Semantic Role Labeling: Semantic role labeling clarifies the functions of each component in a sentence by identifying the semantic roles of words in a text. It identifies key information such as "who did what", "what was done" and "when", helping the system understand the performer of the action, the affected party, the action itself, and related time, place and other elements.

[0034] S25. Contextual relationship analysis and topic modeling: Contextual relationship analysis and topic modeling help the system understand the overall structure and semantic flow of the text by analyzing the logical relationship between sentences and paragraphs in the document.

[0035] For step S3, the text information and visual information are jointly embedded and aligned through the deep learning model. The present invention designs a hierarchical information interaction structure and embeds it into the HRNetv2 network to obtain the following: Figure 2 The explicit semantic information extraction model shown in Figure 2. This design realizes the information interaction between multi-scale global and local information, thereby improving the extraction quality of explicit semantic information.

[0036] like Figure 2As shown in the figure, the hierarchical information interaction structure contains two layers, of which the first layer consists of four single-scale feature information interaction (IISS) structures, corresponding to the feature maps of four scales extracted by HRNetv2. In each IISS structure, the size of the feature map C (1≤i≤4) is first adjusted by adaptive average pooling operation, and adjusted to 1 / 64, 1 / 16, 1 / 4 and 1 times of the original feature map respectively. Then the adjusted feature map is flattened into a sequence, in which each element corresponds to a pixel in the feature map, and a one-dimensional position encoding vector is introduced to encode the pixel position to model its position information. Next, the elements in the sequence are used as satellite nodes of the star-shaped information interaction structure, and the global average pooling result of the sequence is used as the relay node. The star-shaped information interaction structure is used to interact with the node information to obtain the satellite node representation containing rich context information. After obtaining the node representation after integrating the context information, it is restored to the shape of a two-dimensional feature map in combination with the upsampling operation. After the above interaction process, the multi-channel representation of each pixel in the feature map contains information about the surrounding pixels. According to its corresponding receptive field, it can be seen that the information of different areas in the original document image is also interacted.

[0037] After completing the hierarchical information interaction, the multi-scale feature maps after interaction are spliced, and the spliced ​​feature maps are used for pixel-by-pixel classification to obtain the final segmentation result. At this time, the segmentation result contains explicit semantic information of the document image.

[0038] The corresponding step S3 specifically includes:

[0039] S31. Use HRNetv2 to extract multi-scale features of document images.

[0040] S32. Use the hierarchical information interaction structure to interact with the pixels in the feature map and their context information.

[0041] S33, splicing the feature maps after information interaction, and using the semantic segmentation head to perform pixel-by-pixel classification.

[0042] S34. Concatenate the feature vectors to obtain a higher-dimensional feature vector.

[0043] S35. Use a linear mapping layer to transform the dimension to the original dimension, and adaptively adjust the weight of explicit semantic information and other visual information on each dimension.

[0044] For the answer prediction described in step S4, after global and short-distance information interaction, the present invention embeds a hierarchical prediction module based on the idea of ​​multi-granular reading comprehension, gradually narrowing the search range of the answer to improve the accuracy of answer prediction. Figure 3 As shown, the figure includes a problem encoder and a multi-granularity matching module. The step S4 specifically includes:

[0045] S41, using a question encoder composed of a sequence model to encode the question and three tokens representing different granularities. Further enhance the context information of the question token vector in the question context to obtain an enhanced question token vector representation and a token vector representation of the question at three different granularities. The step S41 specifically includes:

[0046] S411, take the text token sequence output by LayoutLMv2 as input, the sequence includes the question token subsequence {q i |1≤i≤m} and document content token subsequence {w i |≤i≤n}, where the vector representation of the question token is obtained based on the global context.

[0047] S412. In order to perform multi-granular answer queries in the document content according to the question, the present invention introduces three special tokens to represent the representation of the question at different granularities, namely [OPAR] at the document granularity, [OSEN] at the sentence granularity, and [QANS] at the token granularity.

[0048] S413, the question encoder formed by the sequence model encodes the question and the three tokens representing different granularities, and further enhances the context information of the question token vector in the question context. After being processed by the encoder, the enhanced question token vector representation and the token vector representation of the question at three different granularities are obtained, as shown in the following formula:

[0049]

[0050] Among them, m represents the maximum length of the question token sequence. After the data set statistics, the maximum length of the question sequence is set to 30. When the length of the question sequence is less than 30, a special token [PAD] is used to fill it. i (1≤i≤m) represents the token in the question sequence, h i (1≤i≤m) represents the vector representation of the question token after the encoder extracts features. Encoder represents the model used to extract sequence features, such as RNN, LSTM, etc.

[0051] S42, using these vector representations of different granularities to perform matching with the document content at three different granularities. Figure 4 As shown in Figure 1, paragraph-level document matching is used to determine whether there is an answer corresponding to the question in the current token sequence.

[0052] S421, each sequence is called a paragraph of the original document, and matching is performed at the paragraph granularity. The specific matching process is shown in the following formula:

[0053]

[0054] In the paragraph matching process, the overall representation vector h of the question at the paragraph granularity is used qpar and the global representation of the paragraph h cls Calculate the matching probability, where h cls It is obtained by taking the average value of all token vectors in the paragraph. (par) Represents the learnable parameters in the bilinear mapping function. When predicting, when the matching probability is greater than the threshold, a more fine-grained prediction is made, otherwise it is considered that the paragraph does not contain the answer to the current question, and further answer search for the current paragraph is terminated.

[0055] S422, find the specific sentence where the answer to the question is located. Divide the paragraph into l sentences, and use overlapping windows to avoid the answer being distributed in multiple sentences.

[0056] S423. Use the vector representation of the question at the sentence granularity to match each sentence one by one to obtain the corresponding probability value, and finally select the sentence with the highest probability as the sentence containing the answer.

[0057] S424, use bidirectional LSTM to encode the sentence to obtain the overall representation of the sentence in the text content, and use the representation to perform sentence-level probability calculation to determine the sentence containing the answer. The specific process is shown in the following formula:

[0058]

[0059] Among them, u represents the length of each sentence in the paragraph, s i represents the overall sentence representation vector obtained by bidirectional LSTM, W (sen) represents learnable parameters.

[0060] S425. After sentence-level matching, determine the sentence containing the answer.

[0061] S426: Perform token-level matching to obtain the starting and ending positions of the answer in the sentence.

[0062] S427: The representation of the question at the token granularity is used as the query vector, and the attention mechanism is used to calculate each token in the question. The obtained attention score is used to perform weighted summation on each token in the question to obtain the token granularity representation that integrates the question information.

[0063] S428: Based on the representation, match each token in the sentence to predict the starting position and ending position of the answer segment. The specific process is shown in the following formula:

[0064]

[0065] in, It represents the token-level representation of the question that further integrates the question information. (a) represents learnable parameters.

[0066] S43, based on the token granularity matching process, the starting and ending positions of the answer fragment are predicted using a bilinear layer, as shown in the following formula:

[0067]

[0068] Among them, W (start) and W (end) is the learnable parameter of the bilinear layer, p (start) and p (end) These are the probability distributions of the token in the sentence as the start and end positions of the answer fragment. The answer fragment is finally obtained by selecting the fragment between tokens corresponding to the maximum probability.

[0069] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A document processing method for a document visual question answering task based on multi-granularity prediction, characterized in that: The following steps are involved: S1. Document format recognition and information extraction: Identify the format of input documents and extract text and image information from them. S2, semantic understanding and context analysis: Natural language processing technology is used to perform semantic understanding and context analysis on text information in documents. S3. Embedding of text and visual information: Jointly embed and align text and visual information through deep learning models; S4. Answer prediction: Based on the questions entered by the user, the answers to the questions are generated by combining text and visual information.

2. The document format recognition and information extraction method according to claim 1, wherein: The step S1 specifically includes: S11, input document format recognition, receiving and identifying the format type of the input document, and determining its format through document metadata or file extension. S12. Text information extraction: For text files (PDF, Word), use appropriate text parsing tools (PyPDF2, python-docx) to extract the text content in the document and retain its structural information. If it is a scanned document or a picture format document, use OCR (optical character recognition) technology to extract the text information. S13. Image information extraction: for documents containing images, charts, or embedded graphics, all image parts in the document are extracted.

3. The semantic understanding and context analysis as claimed in claim 1, characterized in that: The step S2 specifically includes: S21. Text segmentation and annotation: Use the word segmentation algorithm to segment the text, divide the sentences into separate words or phrases, perform part-of-speech tagging on the words in the text, and mark the grammatical role of each word. S22, named entity recognition, identifies and classifies named entities in the text, and uses a pre-trained NER model to identify and extract entities with practical significance. S23, syntactic analysis and dependency analysis, use syntactic analysis and dependency analysis to analyze the sentence structure and the dependency relationship between words in the text, and clarify the logical relationship between words in the sentence. S24, semantic role labeling, uses semantic role labeling technology to identify the semantic roles of different words in sentences in the text, such as who did what, what was done, and when. S25, contextual relationship analysis and topic modeling, based on contextual analysis, understand the logical relationship between sentences and paragraphs in the document, and identify the topics and subtopics in the document. S26, text summarization and key information extraction, through automated text summarization technology, extract key information from documents or generate concise summaries.

4. The embedding of text and visual information as claimed in claim 1, characterized in that The step S3 specifically includes: S31. Attention mechanism. Based on the joint embedding, the attention mechanism is used to perform weighted fusion of text and image features. S32, multimodal fusion and joint reasoning, after the multimodal features are aligned, the multimodal information is further fused, and a deep neural network is used to learn the joint representation of text and image to generate the final multimodal embedding representation. S33, feature integration and semantic reasoning, based on the multimodal features after joint embedding, semantic reasoning is performed to combine the information of text and image to generate the final answer. In this step, the embedded features of text and image will be further integrated through a deep reasoning network to generate answers related to user questions.

5. The answer prediction according to claim 1, characterized in that: The step S4 specifically includes: First, the system uses a multi-granularity matching method to analyze different levels of the question (such as vocabulary level, sentence level, and paragraph level). The matching at each level not only focuses on the literal similarity between the question and the answer, but also considers higher-level semantic associations. At the vocabulary level, the system identifies the direct match between the key terms in the question and the corresponding information in the answer. At the sentence level, through syntactic and semantic analysis, the system evaluates the consistency of the grammatical structure and semantics between the question and the answer. At the paragraph level, the system further considers the overall context and understands the relationship between the question and the answer in the larger text context.

Images