Intelligent question-answering system and method based on multi-modal document view
By combining multi-format document view analysis, retrieval, and question-answering modules, the complexity of modeling different document types in traditional question-answering systems is solved, achieving unified processing and efficient answer return for multiple document types.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN INST OF TECH
- Filing Date
- 2022-09-13
- Publication Date
- 2026-05-22
AI Technical Summary
Traditional open-domain question answering systems struggle to model different document view layouts uniformly and ignore non-plain text information in documents, resulting in single and inefficient answer returns.
The multi-form document view analysis module extracts text information using OCR technology, the multi-form document view retrieval module performs semantic matching and sorting, and the multi-form document view question answering module extracts high-priority document views and returns the answers.
It achieves unified processing of multiple document types, improves the accuracy and efficiency of answers, and can return visual answers, making it suitable for intelligent question-and-answer applications with multi-format document views.
Smart Images
Figure CN115658856B_ABST
Abstract
Description
Technical Field
[0001] It involves the field of natural language processing, specifically document view retrieval. Background Technology
[0002] With the continuous development of natural language processing technology, the capabilities of intelligent question-answering systems are also constantly making breakthroughs. Traditional search engines return a collection of documents based on keywords entered by the user, failing to meet the user's exact needs. The emergence of intelligent question-answering systems aims to directly return the answers users require, eliminating the need for users to search manually, thus allowing them to interact with the system in a natural way. Therefore, how to accurately match documents related to the question and extract the corresponding answers has always been one of the most pressing problems to be solved in the field of question-answering systems.
[0003] Document-view question answering (DRQA) is the process of finding relevant document views from a document library to answer a user's question and accurately locating the answer. It involves multiple fields and can be expanded, offering significant application value, such as analyzing corporate documents to improve office efficiency. Current open-domain question answering systems employ different text extraction methods for different document types based on predetermined rules. Furthermore, most of these systems interact with information in plain text format, parsing user questions and then querying unstructured document libraries to return relevant answers, such as the open-source open-domain question answering system DRQA. However, real-world information is diverse; documents include not only plain text but also visual information such as images, PDFs, and web pages. Therefore, current intelligent question answering systems neglect other heterogeneous information such as tables, images, and spatial layouts, resulting in relatively singular answers that are mostly continuous plain text, lacking universality, efficiency, and accuracy.
[0004] In summary, it can be seen that traditional open-domain question answering systems still have the following problems in the process of extracting question-related answers from different types of documents:
[0005] Different text extraction methods need to be designed for different types of documents, which increases the complexity of the project and makes it lack versatility.
[0006] Question-answering systems often target plain text, thus ignoring other information in the document, such as text appearance images, tables, spatial layout, etc.
[0007] The complex layout of different document views makes it difficult to model all objects uniformly. At the same time, traditional systems struggle to accurately extract answers when the answers to questions are discontinuous. Summary of the Invention
[0008] To address the problem of complex layouts for different types of document views in existing open-domain question-answering systems, making it difficult to model all objects uniformly, the technical solution provided by this invention is as follows:
[0009] The intelligent question-answering system based on multi-form document views includes: a multi-form document view analysis module, a multi-form document view retrieval module, and a multi-form document view question-answering module.
[0010] The multi-form document view analysis module is used to extract text information;
[0011] The multi-form document view retrieval module is used to retrieve document views in the text that are related to preset information and to prioritize them.
[0012] The multi-form document view question-and-answer module is used to extract and output the document views with higher priority.
[0013] Furthermore, a preferred embodiment is provided, wherein the text information includes: text content, text appearance image, and text layout.
[0014] Furthermore, a preferred embodiment is provided in which the multi-form document view analysis module extracts the text information by means of an OCR tool.
[0015] Furthermore, in a preferred embodiment, the multi-form document view analysis module is also used to save the text information.
[0016] Furthermore, a preferred embodiment is provided in which the multi-form document view retrieval module retrieves text by means of a semantic matching method.
[0017] Furthermore, a preferred embodiment is provided, wherein the multi-form document view retrieval module is specifically used for: including:
[0018] The function of retrieving a collection of documents related to preset information from text information, and the function of matching the document view in the document collection with the preset information and sorting them by relevance.
[0019] Based on the same inventive concept, the present invention also provides an intelligent question-answering method based on multi-form document views, the method comprising: a multi-form document view analysis step, a multi-form document view retrieval step, and a multi-form document view question-answering step;
[0020] The multi-form document view analysis steps are as follows: extracting text information;
[0021] The multi-form document view retrieval step is as follows: retrieve document views in the text that are related to preset information and prioritize them;
[0022] The multi-form document view question-and-answer step is as follows: extract the document views with higher priority and output them.
[0023] Furthermore, a preferred embodiment is provided, wherein the multi-form document view retrieval step specifically includes:
[0024] The steps include retrieving a set of documents from text information that are related to preset information, and matching the document views in the document set with the preset information and sorting them by relevance.
[0025] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program, wherein when the computer's processor processes the computer program stored in the storage medium, the computer executes the intelligent question-answering method based on multi-form document views.
[0026] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium, the storage medium being used to store a computer program, characterized in that, when the processor of the computer processes the computer program stored in the storage medium, the computer executes the intelligent question-answering method based on multi-form document views.
[0027] The advantages of this invention are:
[0028] The intelligent question-answering system based on multi-format document views provided by this invention uses OCR technology to perform unified preprocessing on documents of different formats, extracting layout information of text and images, thus overcoming the limitation of traditional systems that require additional extraction algorithms for different types of documents.
[0029] Suitable for use in intelligent question-answering applications with multi-format document views. Attached Figure Description
[0030] Figure 1 This is a schematic diagram of the system flow of the intelligent question-answering system based on multi-format document views mentioned in Implementation Method 1;
[0031] Figure 2 This is a schematic diagram of the semantic matching model based on cross-modal information interaction mentioned in Implementation Method 4;
[0032] Figure 3 This is a schematic diagram of the layout structure mentioned in Implementation Method 4;
[0033] Figure 4 This is a schematic diagram of the global attention interaction layer structure mentioned in Implementation Method 4;
[0034] Figure 5 This is a schematic diagram of a system test example mentioned in Implementation Method Eleven. Detailed Implementation
[0035] To make the advantages and benefits of the technical solution provided by the present invention clearer, the technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings, specifically:
[0036] Implementation Method 1: Combination Figure 1 This embodiment describes an intelligent question-answering system based on multi-form document views. The system includes: a multi-form document view analysis module, a multi-form document view retrieval module, and a multi-form document view question-answering module.
[0037] The multi-form document view analysis module is used to extract text information;
[0038] The multi-form document view retrieval module is used to retrieve document views in the text that are related to preset information and to prioritize them.
[0039] The multi-form document view question-and-answer module is used to extract and output the document views with higher priority.
[0040] Specifically:
[0041] It includes a multi-format document view analysis module, a multi-format document view retrieval module, and a multi-format document view question-and-answer module. For a user's question, the multi-format document view analysis module first uses OCR tools to extract information such as text content, text appearance images, and layout; then, the retrieval module uses text semantic matching technology to retrieve document views related to the question and prioritizes them; finally, the question-and-answer module extracts the answer from the higher-priority document views and returns it to the user along with the document. The specific process is as follows: Figure 1 As shown.
[0042] Implementation Method 2: This implementation method further defines the intelligent question-answering system based on multi-form document view provided in Implementation Method 1. The text information includes: text content, text appearance image, and text layout.
[0043] Implementation Method 3: This implementation method further defines the intelligent question-answering system based on multi-form document view provided in Implementation Method 1 or 2. The method by which the multi-form document view analysis module extracts the text information is: extraction is performed using an OCR tool.
[0044] Implementation Method Four: This implementation method further defines the intelligent question-answering system based on multi-form document view provided in Implementation Method Three. The multi-form document view analysis module is also used to save the text information.
[0045] In the analysis module of the multi-form document view, the information in the multi-form document view is preprocessed by OCR technology to obtain the text content, text appearance image, and document view layout structure information, and save them for subsequent use of image processing technology and natural language processing technology for question answering tasks.
[0046] In the multi-format document view retrieval module, to improve the efficiency of retrieving documents related to user questions, a retrieval + sorting approach is used. First, a preliminary set of documents related to the question is retrieved from a large number of documents based on recall. Second, the returned document set is matched with the question based on precision, and these documents are scored and sorted, returning the document set with the highest relevance. Through retrieval and sorting, the document view most relevant to the user's answer can be obtained.
[0047] In the retrieval module, this invention proposes a semantic matching model based on cross-modal information interaction, SLMBON (Semantic Collocation Model Based on Cross-modal Information Interaction), which mainly consists of a multimodal information embedding layer, a semantic information encoding layer, a multi-form document view and question interaction layer, and a semantic relationship prediction output layer, such as... Figure 2 As shown.
[0048] The multimodal information embedding layer concatenates the document view information obtained from the multi-morphological document view analysis module and the user-submitted question information, transforming them into a unified vector representation. The document view information includes a one-dimensional text sequence, a text appearance image sequence, and a two-dimensional document view layout structure. The one-dimensional text sequence and user question sequence can be directly represented using a word embedding layer, as shown in Equation 4-1.
[0049]
[0050] in, The embedding information represents the text content. TokenEmbed represents the word embedding information of the word itself, PosEmbed represents the word position embedding information, and SegEmbed is used to distinguish different texts.
[0051] The image sequence of the text is first used for feature extraction by a convolutional neural network, and then compressed into a one-dimensional sequence like the text, as shown in Equation 4-2.
[0052]
[0053] in, Embedded information representing the text's appearance image. It is the initial vector representation of all characters, Conv is the convolution operation, and flatten transforms the vector into one dimension.
[0054] Layout structure features are modeled using word coordinates and the distance between words and the border. Word coordinates are obtained by calculating the absolute position of the word and its relative position to other words, while the distance between the word and the border is represented by their absolute distance. The layout structure is visually represented as a rectangle, whose position can be represented by the coordinates of its top-left and bottom-right corners. The height and width of the text box can be expressed as follows: and ,like Figure 3 As shown.
[0055] Therefore, the embedded information of the layout structure can be represented as:
[0056]
[0057] in, This indicates that the embedding information of the x-coordinate and y-coordinate of the above layout structure is spliced together. This indicates that the two-dimensional position embedding information of the document view layout structure has been obtained;
[0058] The semantic information encoding layer uses a multi-head self-attention mechanism to jointly model input information such as multi-form document views and user questions. Specifically, it uses the text content itself, the text appearance image, coordinates, and border distance as attention heads for information interaction.
[0059]
[0060] in, This is a multi-head self-attention mechanism. , , These represent the information of the three attention heads respectively.
[0061] Then, after passing through the gated connection layer, feedforward sub-layer, and residual connection, and undergoing layer normalization, the output is obtained, yielding relevant contextual semantic information. The gated connection layer is an optimization of the residual connection; it no longer simply concatenates the output with the initial input. The specific calculation formula is as follows:
[0062]
[0063] in, (W represents the parameters of a trainable neural network, b represents the bias, and d represents the dimension.) , Let I and O represent the linear mapping results of the initial input I and the output O after passing through the attention layer, respectively. This represents the output after performing forward computation on I and O in a neural network and activating it using the sigmoid function. This represents the final output of the gate connection layer.
[0064] The feedforward sublayer references the Transformer model and uses a feedforward neural network with activation functions, as shown in formula 4-8.
[0065]
[0066] in, It is an activation function. (W are all trainable parameters, b is the bias, and d represents the dimension of the input x). It is the dimension of the hidden layer.
[0067] Finally, the semantic relationship prediction output layer determines the semantic relationship between the user question and the multi-form document view based on the output of the global attention interaction layer. The global attention interaction layer is used to store, update, and transmit semantic interaction information, and its structure diagram is shown below. Figure 4 As shown.
[0068] The global attention interaction layer first calculates the interaction matrix between the multi-form document view information representation and the user question information representation of the current layer, and then updates the global information by concatenating the current interaction matrix with the historical interaction matrix, as shown in formulas 4-9 and 4-10.
[0069]
[0070] in, and These represent the document view information and user question information for the current layer, respectively. , This represents element-wise multiplication of a vector. Indicates the current layer number. [;] indicates matrix concatenation. This is the activation function. , is the dimension of the vectors in the global attention matrix.
[0071] After the global attention matrix is saved and updated, the interaction vector is converted into attention values through the pooling layer, as shown in formula 4-11.
[0072]
[0073] in, For activation function, n represents The length of the layer interaction vector, , Finally, the attention values are normalized and used as the output of the global attention interaction layer.
[0074] In the multi-modal document view question answering module, the multilingual cross-modal pre-trained language model LayoutXLM is used to extract the answers from the document view, and finally the relevant documents and the highlighted text of the answers are returned to the user.
[0075] In this embodiment, in the initial retrieval part of the document view, a relevant pre-trained model, such as RocketQA, can be used for efficient retrieval.
[0076] In this embodiment, the extraction of text appearance image features can be achieved without using a convolutional neural network. The image can be directly divided into several blocks, each block being similar to a word in the text, and then transformed into one dimension by a linear mapping layer.
[0077] In this embodiment, the cross-modal semantic matching model proposed in this invention can also be used for answer extraction.
[0078] The intelligent question-answering system based on multi-form document views provided in this embodiment offers visually appealing and easy-to-understand answers to user questions by utilizing multi-source heterogeneous document views. The document types come from document collections with different structures such as web pages, PDFs, and Word documents. Through OCR technology, documents of different formats can be preprocessed in a unified manner to extract the layout information of text and images, thus overcoming the limitation of traditional systems that require additional extraction algorithms for different types of documents.
[0079] The cross-modal semantic matching model based on multi-morphic document views provided in this embodiment can simultaneously model text, text appearance images, and spatial layout, thus solving the problem of cross-modal semantic matching.
[0080] This implementation tested the system using 20 data points from different types of document views and achieved the expected results. By inputting a question, the system returns relevant documents and highlights the corresponding answers for easy and intuitive understanding by the user.
[0081] Implementation Method 5: This implementation method further defines the intelligent question-answering system based on multi-form document view provided in Implementation Method 1. The method for the multi-form document view retrieval module to retrieve text is: through semantic matching.
[0082] Implementation Method Six: This implementation method further defines the intelligent question-answering system based on multi-form document views provided in Implementation Method One. The multi-form document view retrieval module specifically includes:
[0083] The function of retrieving a collection of documents related to preset information from text information, and the function of matching the document view in the document collection with the preset information and sorting them by relevance.
[0084] Implementation Method Seven: This implementation method provides an intelligent question-answering method based on multi-form document views. The method includes: a multi-form document view analysis step, a multi-form document view retrieval step, and a multi-form document view question-answering step.
[0085] The multi-form document view analysis steps are as follows: extracting text information;
[0086] The multi-form document view retrieval step is as follows: retrieve document views in the text that are related to preset information and prioritize them;
[0087] The multi-form document view question-and-answer step is as follows: extract the document views with higher priority and output them.
[0088] Implementation Method Eight: This implementation method further defines the intelligent question-answering method based on multi-form document views provided in Implementation Method Seven. The multi-form document view retrieval steps specifically include:
[0089] The steps include retrieving a set of documents from text information that are related to preset information, and matching the document views in the document set with the preset information and sorting them by relevance.
[0090] Implementation Method Nine: This implementation method provides a computer storage medium for storing computer programs. When the computer's processor processes the computer programs stored in the storage medium, the computer executes the intelligent question-answering method based on multi-form document views provided in Implementation Method Seven or Eight.
[0091] Implementation Method 10: This implementation method provides a computer, including a processor and a storage medium, wherein the storage medium is used to store a computer program. When the processor of the computer processes the computer program stored in the storage medium, the computer executes the intelligent question-answering method based on multi-form document view provided in Implementation Method 7 or 8.
[0092] Implementation Method Eleven: This implementation method provides a specific implementation of the intelligent question-answering system based on multi-format document views provided in Implementation Method One, and also serves to explain Implementation Methods One through Six. Specifically:
[0093] An intelligent question-answering system based on multi-format document views: The system specifically includes a multi-format document view analysis module, a retrieval module, and a question-answering module. For a user's question, the analysis module processes document views with heterogeneous information in a unified manner, thus providing a prerequisite for subsequent document view retrieval and document view question-answering. The document view retrieval module searches document views based on the user's question, returns candidate documents with high relevance through scoring and ranking, and finally, the document view question-answering module extracts the answer and returns it to the user.
[0094] In the aforementioned multi-form document view analysis module, OCR technology is used to extract information from the multi-form document view, mainly extracting the text content and the corresponding layout information after word segmentation. Specifically, the absolute position of words, the relative position between words, the text box height of words, and the height difference between text boxes of words are used as features of the layout structure. This achieves the transformation from one-dimensional plain text modeling to two-dimensional spatial structure modeling of the document view.
[0095] In the aforementioned multi-modal view retrieval module, user-submitted questions are used to retrieve and sort document views. This includes coarse-grained retrieval using the BM25 algorithm and fine-grained sorting using the SLMBON semantic matching model based on cross-modal information interaction proposed in this invention. The cross-modal information semantic matching model mainly consists of a multi-modal information embedding layer, a semantic information encoding layer, an interaction layer between multi-modal document views and questions, and a semantic relationship prediction output layer.
[0096] The multimodal information embedding layer aims to uniformly transform text content and its spatial layout information into vector representations. The semantic information encoding layer uses a multi-head self-attention mechanism to jointly encode the text content, text appearance image, text coordinates, and border distance to obtain a vector representation with contextual semantic information. The multi-morphic document view and question interaction layer uses the three-dimensional attention matrix proposed in this invention to maintain global semantic interaction information and obtain the final attention vector representation. Finally, the semantic relationship prediction output layer predicts the semantic relationship between the user's question and the document view based on the probability distribution of the labels.
[0097] In the aforementioned multi-modal document view question answering module, answers need to be extracted from the high-priority documents obtained by the retrieval module. The extraction model uses LayoutXLM, the first publicly available multi-modal document understanding pre-trained model, combined with CRF for sequence labeling. After jointly modeling and pre-training the text content, text appearance image, and document view layout information in the document view, it is applied to the downstream task of document view question answering to extract answers. This model can extract discontinuous answer sequences and return the results, preserving the source document layout, to the user for intuitive understanding.
[0098] After implementing each module of the above-described multi-format document view intelligent question-and-answer system, this implementation method conducted an example test on the system.
[0099] The following is an example of a complete system process, such as a user inputting the question: "How do I zoom in and out of pictures on my phone?".
[0100] First, before the system answers a question, it performs OCR parsing on the documents in the document library and saves the results for the retrieval module. The user's question is then passed to the retrieval module, which searches and sorts relevant documents. The results are then sent to the question-and-answer module, which extracts the answer that matches the question and returns it to the user. The answer is as follows: Figure 5 As shown.
[0101] The above description of several specific embodiments further details the technical solution provided by the present invention in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any modifications and improvements to the present invention, combinations and substitutions of embodiments based on the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An intelligent question-answering system based on multi-format document views, characterized in that, The system includes: a multi-form document view analysis module, a multi-form document view retrieval module, and a multi-form document view question and answer module; The multi-form document view analysis module is used to extract text information; The multi-form document view retrieval module is used to retrieve document views in the text that are related to preset information and to prioritize them. The multi-form document view question-and-answer module is used to extract and output the document views with higher priority. Specifically: In the retrieval module, the semantic matching model SLMBON based on cross-modal information interaction is divided into a multimodal information embedding layer, a semantic information encoding layer, a multi-form document view and question interaction layer, and a semantic relationship prediction output layer. The multimodal information embedding layer concatenates the document view information obtained from the multi-morphological document view analysis module and the user-submitted question information, transforming them into a unified vector representation. The document view information includes a one-dimensional text sequence, a text appearance image sequence, and a two-dimensional document view layout structure. Here, the one-dimensional text sequence and the user question sequence are directly represented using a word embedding layer. in, The embedding information represents the text content; TokenEmbed represents the word embedding information of the word itself; PosEmbed represents the word position embedding information; and SegEmbed is used to distinguish different texts. The image sequence is first processed by a convolutional neural network to extract features, and then compressed into a one-dimensional sequence similar to text. in, Embedded information representing the text's appearance image. It is the initial vector representation of all characters, Conv is the convolution operation, and flatten transforms the vector into one dimension; The layout structure features are modeled using word coordinates and the distance between words and borders. Word coordinates are obtained by calculating the absolute position of the word and the relative position between words. The distance between a word and the border is represented by the absolute distance between them. The embedded information of the layout structure is represented as follows: in, This indicates that the embedding information of the x-coordinate and y-coordinate of the above layout structure is spliced together. This indicates that the two-dimensional position embedding information of the document view layout structure has been obtained. The coordinates are represented by the top-left and bottom-right corners. and The height and width of the text box; The semantic information encoding layer utilizes a multi-head self-attention mechanism to jointly model multi-form document views and user question input information: in, This is a multi-head self-attention mechanism. , , These represent the information of the three attention heads respectively; The global attention interaction layer first calculates the interaction matrix between the multi-form document view information representation and the user question information representation of the current layer, and then updates the global information by concatenating the current interaction matrix with the historical interaction matrix. in, and These represent the document view information and user question information for the current layer, respectively. , This represents element-wise multiplication of a vector. Indicates the current layer number; [;] indicates matrix concatenation. For activation function, , is the dimension of the vectors in the global attention matrix; After saving and updating the global attention matrix, the interaction vectors are transformed into attention values through a pooling layer: in, For activation function, n represents The length of the layer interaction vector, , Finally, the attention values are normalized and used as the output of the global attention interaction layer. In the multi-modal document view question answering module, the multilingual cross-modal pre-trained language model LayoutXLM is used to extract the answers from the document view, and finally the relevant documents and the highlighted text of the answers are returned to the user.
2. The intelligent question-answering system based on multi-format document views according to claim 1, characterized in that, The text information includes: text content, text appearance image, and text layout.
3. The intelligent question-answering system based on multi-format document views according to claim 1 or 2, characterized in that, The method by which the multi-form document view analysis module extracts the text information is: extraction using an OCR tool.
4. The intelligent question-answering system based on multi-format document views according to claim 3, characterized in that, The multi-form document view analysis module is also used to save the text information.
5. The intelligent question-answering system based on multi-format document views according to claim 1, characterized in that, The method for retrieving text in the multi-form document view retrieval module is through semantic matching.
6. The intelligent question-answering system based on multi-format document views according to claim 1, characterized in that, The multi-form document view retrieval module is specifically used for: including: The function of retrieving a collection of documents related to preset information from text information, and the function of matching the document view in the document collection with the preset information and sorting them by relevance.
7. An intelligent question-answering method based on multi-format document views, characterized in that, The method is implemented based on the system described in claim 1, and includes: a multi-form document view analysis step, a multi-form document view retrieval step, and a multi-form document view question-and-answer step; The multi-form document view analysis steps are as follows: extracting text information; The multi-form document view retrieval step is as follows: retrieve document views in the text that are related to preset information and prioritize them; The multi-form document view question-and-answer step is as follows: extract the document views with higher priority and output them.
8. The intelligent question-answering method based on multi-form document views according to claim 7, characterized in that, The multi-format document view retrieval steps specifically include: The steps include retrieving a set of documents from text information that are related to preset information, and matching the document views in the document set with the preset information and sorting them by relevance.
9. A computer, comprising a processor and a storage medium, said storage medium for storing computer programs, characterized in that, When the processor of the computer processes the computer program stored in the storage medium, the computer executes the intelligent question-answering method based on multi-morphological document views as described in any one of claims 7-8.
10. A computer storage medium for storing computer programs, characterized in that, When the computer's processor processes the computer program stored in the storage medium, the computer executes the intelligent question-answering method based on multi-morphological document views as described in any one of claims 7-8.