Word document question and answer method and system based on large language model

Abstract

The invention discloses a word document question and answer method and system based on a large language model, and belongs to the technical field of artificial intelligence. The method comprises the steps that when a document is preprocessed, the position of picture content in an original document is recorded, and after the picture content is described and interpreted through a multi-modal model, the position of the picture content in the original document is recorded; inserting an interpretation result returned by the multi-modal model into an original position of the picture; in the dialogue stage, different processing mechanisms are adopted according to the length of a document: when the document is a long document with the token number exceeding a threshold value, the document is partitioned and vectorized and then stored in a vector database, and an RAG mechanism is started; and for the short-length documents of which the document token number does not reach the threshold value, performing question and answer in a block traversing manner. According to the invention, the accuracy and user experience of the document question-answering system can be improved; and unnecessary vectorization and database storage of the short documents can be avoided, and the processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a word document question-answering method and system based on a large language model. Background Art

[0002] The existing technology pre-processes the Word document, saves the chart in the document as a single image, and then uses a multimodal model to interpret the image. The image analysis content is not restored to the position in the original document, that is, the text and image are processed separately, which changes the order of the original document's text and image content.

[0003] Large multimodal language models have made significant progress in processing multimodal data such as images and text. They can describe and interpret images, and incorporate contextual information to provide more accurate interpretations. However, large multimodal models typically have a large number of parameters and high computational resource requirements, which can lead to slow processing speeds. For example, when processing complex documents, problems such as slow speed, low accuracy, and poor compatibility still exist.

[0004] Retrieval-augmented generation (RAG) technology has been widely used in the field of natural language processing. By vectorizing user questions, searching for relevant documents in a vector database, and then combining it with a large model to generate answers, it effectively improves the accuracy and efficiency of question answering. However, RAG technology faces many challenges in building a knowledge base, especially when parsing complex documents, which can lead to information loss or structural disorganization. Summary of the Invention

[0005] The technical task of the present invention is to address the above shortcomings and provide a word document question-answering method and system based on a large language model, which can improve the accuracy and user experience of the document question-answering system; and avoid unnecessary vectorization and database storage of short documents, thereby improving processing efficiency.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A word document question answering method based on a large language model, the implementation of which includes:

[0008] When preprocessing a document, the position of the image content in the original document is recorded. After using the multimodal model to describe and interpret the image content, the interpretation result returned by the multimodal model is inserted into the original position of the image, so that the preprocessed content is consistent with the order of the original document, thereby maintaining the original semantic coherence of the text and images in the document.

[0009] During the dialogue phase, different processing mechanisms are adopted according to the document length: when the document is a long document with the number of tokens exceeding the threshold, the document is divided into blocks, vectorized, and stored in the vector database, and the RAG (retrieval augmentation generation) mechanism is enabled; when the user asks a question, the question is vectorized, and the document content related to the question is retrieved from the vector database, and the answer is generated based on the search results; this mechanism can efficiently process long documents and improve the accuracy of retrieval and answering; for short documents with the number of document tokens below the threshold, a traversal block method is used for question and answer, in which the document content is divided into blocks, and the relevance of each block to the user's question is calculated. Only blocks with higher relevance are processed and answers are generated; finally, the answers to each block are aggregated and summarized to form the final answer.

[0010] Furthermore, when preprocessing the document, the position of the image content in the original document is recorded, and the original document content and semantic coherence are maintained through special image marking and replacement; the specific implementation is as follows:

[0011] When a user uploads a Word document containing images, a unique number is generated for each image and saved as an independent file. At the same time, the image is replaced with a special tag in the original document. This mechanism not only retains the image's location information in the document, but also reserves space through the tag for inserting descriptions and interpretations of the image. In subsequent processing, the image's description information is accurately inserted into the tag position, thereby maintaining the content and semantic coherence of the original document.

[0012] Furthermore, the special mark design of the picture is as follows:

[0013] [PIC_PLACEHOLDER_START:PIC_001]

[0014] {{PIC_001_DESC}}

[0015] [PIC_PLACEHOLDER_END:PIC_001].

[0016] Furthermore, the specific process of preprocessing the document is as follows:

[0017] (1) The user opens a new session and uploads a word document in the dialogue interface. The system preprocesses the received word document: first, the document structure is parsed. Python-docx can be used to traverse the document. When traversing the document, for each picture encountered, it is numbered and saved as an image file. At the same time, the picture is replaced with a special tag in the document. The purpose of the tag is to facilitate the reinsertion of the content of the picture after being interpreted by the multimodal large language model to its original position, so as to restore the content order of the original document; in addition, the preceding and following paragraphs of the picture (usually the name of the picture in the middle is taken, such as "Figure 1 The text content of the image (e.g., ".xx schematic diagram") is stored as a file named "image number.txt" to be used as auxiliary context information when calling the multimodal model to interpret the image. When a table is encountered, it is converted into a markdown table format to facilitate the subsequent transmission of the data information in the table to the large language model.

[0018] (2) Use the multimodal large language model to describe and interpret the image file. When organizing the prompt words of the multimodal model, the "image number.txt" file will be read. By combining the context text of the paragraph where the image is located (especially the previous and next paragraphs), rich semantic information is provided to the multimodal model, thereby helping the model to better understand the content and meaning of the image. Since the image has been numbered and saved as a file in step (1), the processing here can be parallelized according to the performance of the multimodal large model.

[0019] (3) inserting the description and interpretation information of the image into the special mark reserved for it in the original document according to its number, replacing the placeholder description part in the special mark, so that the document can maintain the original content order;

[0020] (4) After the above steps, the images and tables in the document passed to this session are converted into text form, and the total token number of the document is calculated. If the number of document tokens exceeds the specified threshold, it is divided into blocks, vectorized, and stored in the vector database, and the variable RAG_ENABLE is set to True; if the number of document tokens does not reach the specified threshold, no processing is performed, and only the variable RAG_ENABLE is set to False.

[0021] Furthermore, the specific implementation process of the dialogue stage is as follows:

[0022] After document preprocessing is complete, users can engage in a question-and-answer conversation with the system regarding the document content. The user sends a question and the system checks the value of the current session variable RAG_ENABLE. If RAG_ENABLE = True, the RAG (Retrieval Enhancement Generation) process is executed. After the user's question is vectorized, the vector database is searched for document content related to the user's question. The document content, together with the user's question, forms a prompt word for the large model, which then processes the question and provides an answer to the user's question.

[0023] If RAG_ENABLE=False, it means that the document content is short. First, the document content is divided into blocks, and a document question-and-answer dialogue is conducted by traversing each block. Before each block dialogue, the relevance of the current block to the user question is calculated. If the relevance reaches the threshold, the large model dialogue for the block is executed. If the relevance does not reach the threshold, it is skipped. After completing the entire document traversal, the answers to the user questions in each block are summarized and summarized by the large model to form the document's final answer to the user question.

[0024] Furthermore, during document preprocessing, the Python-docx library is used to implement document reading and traversal, image extraction and tag replacement, and table conversion to Markdown format.

[0025] Furthermore, the document is read and traversed:

[0026] Python-docx provides the function of reading Word documents. It can load documents and traverse their contents paragraph by paragraph and element by element. By loading the document object, you can access elements such as paragraphs and tables in the document. Paragraphs are the basic units of document content, and each paragraph object can extract its text content; tables can be accessed through the structure of rows and cells, and users can extract data from tables row by row and cell by cell. This function enables the system to fully parse the document content, providing a basis for subsequent processing steps.

[0027] The image extraction and tag replacement:

[0028] When processing images in documents, Python-docx can identify image elements in the document and obtain their location information in the document. Although Python-docx itself does not directly support extracting image data, it can use the image placeholder information it provides in combination with other tools (such as file operation libraries or specialized image processing tools) to extract images and save them as independent files. In addition, Python-docx also supports inserting custom markup content into documents, such as inserting specific placeholder tags at the image location. These tags can be used to subsequently reinsert the description and interpretation of the image into the document, thereby maintaining the integrity and semantic coherence of the document content.

[0029] The table is converted to Markdown format:

[0030] Python-docx provides detailed operations on tables within documents, allowing users to access table rows and cells and extract data content. This allows for converting table data into other formats, such as Markdown. Markdown is a lightweight markup language that is concise and easy to read, making it suitable for subsequent text processing and presentation. By converting tables to Markdown, the system can better pass the table data to large language models for analysis and processing, while also facilitating subsequent document content integration and presentation.

[0031] The present invention also claims protection for a Word document question-answering system based on a large language model, comprising:

[0032] The image tagging module is used to record the position of the image content in the original document during the document preprocessing stage. After using the multimodal model to describe and interpret the image content, the interpretation result returned by the multimodal model is inserted into the original position of the image, so that the preprocessed content is consistent with the original document sequence, thereby maintaining the original semantic coherence of the text and images in the document;

[0033] The answer generation module is used to adopt different processing mechanisms based on document length during the dialogue phase: when a document is long and the number of tokens exceeds a threshold, the document is divided into blocks, vectorized, and stored in a vector database, and the RAG (retrieval-augmented generation) mechanism is enabled. When a user asks a question, the system vectorizes the question, searches the vector database for document content related to the question, and generates an answer based on the search results. This mechanism can efficiently process long documents and improve the accuracy of retrieval and answers. For short documents whose document token number does not reach the threshold, a traversal block approach is used for question and answer, dividing the document content into blocks, calculating the relevance of each block to the user's question, and only processing the blocks with high relevance to generate answers. Finally, the answers from each block are aggregated and summarized to form the final answer.

[0034] The system specifically implements word document question and answer through the above method.

[0035] The present invention also claims a large language model-based word document question-answering device, comprising: at least one memory and at least one processor;

[0036] The at least one memory is configured to store a machine-readable program;

[0037] The at least one processor is configured to call the machine-readable program to implement the above method.

[0038] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which are capable of implementing the above method when executed by a processor.

[0039] Compared with the prior art, the word document question-answering method and system based on a large language model of the present invention has the following beneficial effects:

[0040] 1. Traditional document processing methods often ignore or delete images, resulting in incomplete document content. For example, some simple document parsing tools only extract textual content, ignoring images and their contextual information. Furthermore, in some multimodal document question-and-answer systems, image processing is often crude, failing to accurately restore the image's position and semantic relationship within the document. This invention effectively addresses these issues through a special tagging and replacement mechanism, improving the accuracy and user experience of the document question-and-answer system.

[0041] 2. Traditional document question-answering systems typically apply a uniform processing approach to all documents, regardless of length. For example, some systems directly pass the entire document content to the language model, which leads to inefficiency and waste of resources when processing long documents. Other systems, while using the RAG mechanism, fail to optimize for document length, resulting in excessive complexity when processing short documents. This invention significantly improves system performance by dynamically selecting a processing mechanism based on document length, balancing efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a diagram illustrating a document preprocessing process provided by an embodiment of the present invention;

[0043] Figure 2 It is a diagram illustrating the dialogue processing process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below with reference to specific embodiments.

[0045] An embodiment of the present invention provides a word document question answering method based on a large language model. The implementation of the method includes:

[0046] When preprocessing a document, the position of the image content in the original document is recorded. After using the multimodal model to describe and interpret the image content, the interpretation result returned by the multimodal model is inserted into the original position of the image, so that the preprocessed content is consistent with the order of the original document, thereby maintaining the original semantic coherence of the text and images in the document.

[0047] During the dialogue phase, different processing mechanisms are adopted according to the length of the document. When the document is long (the number of tokens exceeds the threshold), the document is divided into blocks, vectorized, and stored in the vector database, and the RAG (retrieval enhanced generation) mechanism is enabled. When the user asks a question, the question is vectorized, and the document content related to the question is retrieved from the vector database, and the answer is generated based on the search results. This mechanism can efficiently process long documents and improve the accuracy of retrieval and answers. For shorter documents (the number of tokens does not reach the threshold), a traversal block method is used for question and answer. The document content is divided into blocks, and the relevance of each block to the user's question is calculated. Only the blocks with higher relevance are processed and answers are generated. Finally, the answers to each block are aggregated and summarized to form the final answer.

[0048] Through this large-scale model-based Word document question-and-answer solution, users can upload their own Word documents through the system's dialogue interface and then engage in a document-based question-and-answer dialogue with the system. The key technologies for implementing this method lie in the following two points:

[0049] 1. A mechanism for special marking and replacement of images to maintain the content and semantic coherence of the original document.

[0050] This method proposes a mechanism for processing images in a document question-answering system. When a user uploads a Word document containing images, the system generates a unique number for each image and saves it as a separate file. Simultaneously, the image is replaced in the original document with special markers (such as [PIC_PLACEHOLDER_START:PIC_001], {{PIC_001_DESC}}, and [PIC_PLACEHOLDER_END:PIC_001]). This mechanism not only preserves the image's position within the document but also reserves space for inserting a description and interpretation of the image through the markers. In subsequent processing, the system accurately inserts the image description information at the marker location, maintaining the content and semantic coherence of the original document.

[0051] Traditional document processing methods often ignore or delete images, resulting in incomplete document content. For example, some simple document parsing tools only extract textual content, ignoring images and their contextual information. Furthermore, in some multimodal document question-answering systems, image processing is often crude, failing to accurately restore the image's position and semantic relationship within the document. This method effectively addresses these issues through a special tagging and replacement mechanism, improving the accuracy and user experience of document question-answering systems.

[0052] 2. During the dialogue phase, a mechanism is used to process documents of different lengths using RAG or traversal blocks for question and answer.

[0053] During the conversation phase, different processing mechanisms are employed based on document length. When a document is long (the number of tokens exceeds a threshold), the system segments the document into chunks, vectorizes it, and stores it in a vector database. The RAG (Retrieval Enhanced Generation) mechanism is then activated. When a user asks a question, the system vectorizes the question, searches the vector database for relevant documents, and generates an answer based on the search results. This mechanism efficiently processes long documents and improves the accuracy of both retrieval and answers.

[0054] For shorter documents (those with fewer than the token count threshold), the system uses a block-by-block approach for question answering. The system divides the document content into blocks, calculates the relevance of each block to the user's question, and processes and generates answers only for the blocks with the highest relevance. Finally, the system aggregates and summarizes the responses from each block to form the final answer. This mechanism avoids unnecessary vectorization and database storage for short documents, improving processing efficiency.

[0055] Traditional document question answering systems typically apply a uniform approach to all documents, regardless of length. For example, some systems directly pass the entire document content to the language model, which leads to inefficiency and resource waste when processing long documents. Other systems, while using RAG mechanisms, fail to optimize for document length, resulting in excessive complexity when processing short documents. This approach significantly improves system performance by dynamically selecting a processing mechanism based on document length, balancing efficiency and accuracy.

[0056] The overall operation process of this method includes the following steps:

[0057] 1. Document preprocessing stage.

[0058] (1) The user opens a new session and uploads a word document in the dialogue interface. The system pre-processes the received word document. First, the document structure is parsed and the document is traversed using python-docx. When traversing the document, each picture encountered is numbered and saved as an image file. At the same time, the picture is replaced with the following special mark in the document. The purpose of the mark is to facilitate the reinsertion of the content of the picture after it is interpreted by the multimodal large language model to its original position to restore the content order of the original document. In addition, the previous paragraph and the next paragraph of the picture (usually the picture name in the middle is taken, such as " Figure 1 .xx schematic diagram”) is also stored as a file named “picture number.txt”, which is used as auxiliary context information when calling the multimodal model to interpret the image; when encountering a table, it is converted into a markdown table format to facilitate the subsequent transmission of the data information in the table to the large language model.

[0059] The special mark design of the picture is as follows:

[0060] [PIC_PLACEHOLDER_START:PIC_001]

[0061] {{PIC_001_DESC}}

[0062] [PIC_PLACEHOLDER_END:PIC_001]

[0063] (2) Use a large multimodal language model to describe and interpret the image file. When organizing the prompt words of the multimodal model, the "image number.txt" file will be read. By combining the context of the paragraph where the image is located (especially the previous and next paragraphs), it can provide richer semantic information for the multimodal model, thereby helping the model better understand the content and meaning of the image. Since the images have been numbered and saved as files, the processing here can be parallelized based on the performance of the large multimodal model.

[0064] (3) Insert the description and interpretation information of the picture into the special mark reserved for it in the original document according to its number, replacing the "{{PIC_001_DESC}}" part, so that the document can maintain the original content order.

[0065] (4) After the first three steps, the images and tables in the document passed to this session are converted into text form, and the total token number of the document is calculated. If the document is long (the number of tokens exceeds the specified threshold), it is divided into blocks, vectorized, and stored in the vector database, and the variable RAG_ENABLE is set to True; if the document is short (the number of tokens does not reach the specified threshold), no processing is performed, and only the variable RAG_ENABLE is set to False.

[0066] 2. Dialogue stage.

[0067] After completing document preprocessing, the user can have a question-and-answer conversation with the system about the file content. The user sends a question and detects the value of the variable RAG_ENABLE in the current session. If RAG_ENABLE = True, the RAG (retrieval enhancement generation) process is executed. After the user question is vectorized, the document content related to the user question is retrieved in the vector database, and together with the user question, a prompt word is formed for the large model, which processes the question and gives an answer to the user question.

[0068] If RAG_ENABLE=False, indicating a short document, the document is first divided into blocks. A document question-and-answer dialogue is conducted by traversing each block. Before each block dialogue, the relevance of the current block to the user's question is calculated. If the relevance reaches a threshold, the macro-model dialogue for that block is executed. If the relevance does not reach the threshold, the dialogue is skipped. After the entire document is traversed, the answers to the user's question in each block are summarized and summarized by the macro-model to form the document's final answer to the user's question.

[0069] Python-docx is a powerful Python library specifically designed for working with Microsoft Word documents (.docx format). It allows developers to programmatically read, modify, and generate Word document content, supporting operations on a variety of document elements, including text, paragraphs, tables, and images. This library is widely used in scenarios such as automated document generation, document content extraction, and document format conversion, significantly improving the efficiency and flexibility of document processing.

[0070] In this method, the Python-docx library plays a crucial role in document preprocessing. It not only reads and traverses document content but also supports processing images and tables, providing powerful document parsing and content conversion capabilities for the document question-answering system. These capabilities enable the system to efficiently process uploaded Word documents and provide accurate and coherent document content for subsequent question-answering conversations, as follows:

[0071] 1. Document reading and traversal:

[0072] Python-docx provides functionality for reading Word documents, loading documents and traversing their contents paragraph by paragraph and element by element. By loading a document object, you can access elements within the document, such as paragraphs and tables. Paragraphs are the fundamental unit of document content, and the text content of each paragraph object can be extracted. Tables can be accessed through a row and cell structure, allowing users to extract data from tables row by row and cell by cell. This functionality enables the system to fully parse document content, providing a foundation for subsequent processing steps.

[0073] 2. Image extraction and tag replacement:

[0074] When processing images in documents, Python-docx can identify image elements and retrieve their location information within the document. Although Python-docx itself does not directly support extracting image data, it can use the image placeholder information it provides in conjunction with other tools (such as file manipulation libraries or specialized image processing tools) to extract images and save them as separate files. Furthermore, Python-docx supports inserting custom markup content into documents, such as inserting specific placeholder tags at image locations. These tags can be used to subsequently reinsert image descriptions and interpretations into the document, thereby maintaining the integrity and semantic coherence of the document content.

[0075] 3. Convert the table to Markdown format:

[0076] Python-docx provides detailed operations on tables within documents. Users can access table rows and cells and extract the data within them. This allows for converting table data into other formats, such as Markdown. Markdown is a lightweight markup language that is concise and easy to read, making it suitable for subsequent text processing and presentation. By converting tables to Markdown, the system can better pass the table data to large language models for analysis and processing, while also facilitating subsequent document content integration and presentation.

[0077] An embodiment of the present invention further provides a word document question-answering system based on a large language model, comprising:

[0078] The image tagging module is used to record the position of the image content in the original document during the document preprocessing stage. After using the multimodal model to describe and interpret the image content, the interpretation result returned by the multimodal model is inserted into the original position of the image, so that the preprocessed content is consistent with the original document sequence, thereby maintaining the original semantic coherence of the text and images in the document;

[0079] The answer generation module is used to adopt different processing mechanisms based on document length during the dialogue phase: when a document is long and the number of tokens exceeds a threshold, the document is divided into blocks, vectorized, and stored in a vector database, and the RAG (retrieval-augmented generation) mechanism is enabled. When a user asks a question, the system vectorizes the question, searches the vector database for document content related to the question, and generates an answer based on the search results. This mechanism can efficiently process long documents and improve the accuracy of retrieval and answers. For short documents whose document token number does not reach the threshold, a traversal block approach is used for question and answer, dividing the document content into blocks, calculating the relevance of each block to the user's question, and only processing the blocks with high relevance to generate answers. Finally, the answers from each block are aggregated and summarized to form the final answer.

[0080] The system specifically implements word document question answering through the word document question answering method based on the large language model described in the above embodiment.

[0081] Document preprocessing stage:

[0082] 1. The user opens a new session and uploads a Word document in the conversation interface. The system pre-processes the received Word document. First, it analyzes the document structure and uses python-docx to traverse the document. When traversing the document, for each picture encountered, it is numbered and saved as an image file. At the same time, the picture is replaced with the following special mark in the document. The purpose of the mark is to facilitate the reinsertion of the content of the picture after being interpreted by the multimodal large language model to its original position to restore the content order of the original document. In addition, the preceding and following paragraphs of the picture (usually the name of the picture in the middle is taken, such as " Figure 1 .xx schematic diagram”) is also stored as a file named “picture number.txt”, which is used as auxiliary context information when calling the multimodal model to interpret the image; when encountering a table, it is converted into a markdown table format to facilitate the subsequent transmission of the data information in the table to the large language model.

[0083] The special mark design of the picture is as follows:

[0084] [PIC_PLACEHOLDER_START:PIC_001]

[0085] {{PIC_001_DESC}}

[0086] [PIC_PLACEHOLDER_END:PIC_001]

[0087] 2. Use a large multimodal language model to describe and interpret image files. When the system organizes the multimodal model's prompt words, it reads the "image number.txt" file. By combining the context of the paragraph containing the image (especially the preceding and following paragraphs), it provides the multimodal model with richer semantic information, helping the model better understand the image's content and meaning. Since the images have already been numbered and saved as files, this processing can be parallelized based on the performance of the large multimodal model.

[0088] 3. Insert the description and interpretation information of the picture into the special mark reserved for it in the original document according to its number, replacing the "{{PIC_001_DESC}}" part, so that the document can maintain the original content order.

[0089] 4. After the first three steps, the system converts all images and tables in the document passed in this session into text form and calculates the total token number of the document. If the document is long (the number of tokens exceeds the specified threshold), the system divides it into blocks, vectorizes it, and stores it in the vector database, and sets the variable RAG_ENABLE = True; if the document is short (the number of tokens does not reach the specified threshold), no processing is performed, and only the variable RAG_ENABLE = False is set.

[0090] Dialogue stage:

[0091] After completing document preprocessing, the user can have a question-and-answer conversation with the system about the file content. The user sends a question, and the system detects the value of the variable RAG_ENABLE in the current session. If RAG_ENABLE = True, the RAG (retrieval enhancement generation) process is executed. After the user question is vectorized, the document content related to the user question is retrieved in the vector database, and together with the user question, a prompt word is formed for the large model, which processes the question and gives an answer to the user question.

[0092] If RAG_ENABLE=False, indicating a short document, the document is first divided into blocks. A document question-and-answer dialogue is conducted by traversing each block. Before each block dialogue, the relevance of the current block to the user's question is calculated. If the relevance reaches a threshold, the macro-model dialogue for that block is executed. If the relevance does not reach the threshold, the dialogue is skipped. After the entire document is traversed, the answers to the user's question in each block are summarized and summarized by the macro-model to form the document's final answer to the user's question.

[0093] An embodiment of the present invention further provides a word document question-answering device based on a large language model, comprising: at least one memory and at least one processor;

[0094] The at least one memory is configured to store a machine-readable program;

[0095] The at least one processor is used to call the machine-readable program to implement the word document question-answering method based on the large language model described in the above embodiment.

[0096] An embodiment of the present invention further provides a computer-readable medium having computer instructions stored thereon. When executed by a processor, the computer instructions cause the processor to execute the large language model-based word document question-answering method described in the above embodiments. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above embodiments, and causes a computer (or CPU or MPU) of the system or device to read and execute the program code stored in the storage medium.

[0097] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0098] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0099] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0100] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0101] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A word document question answering method based on a large language model, characterized in that: The implementation of this method includes: When preprocessing a document, the position of the image content in the original document is recorded. After using the multimodal model to describe and interpret the image content, the interpretation result returned by the multimodal model is inserted into the original position of the image, so that the preprocessed content is consistent with the order of the original document, thereby maintaining the original semantic coherence of the text and images in the document. During the dialogue phase, different processing mechanisms are adopted according to the document length: when the document is a long document with the number of tokens exceeding the threshold, the document is divided into blocks, vectorized, and stored in the vector database, and the RAG mechanism is enabled; when the user asks a question, the question is vectorized, and the document content related to the question is retrieved from the vector database, and the answer is generated based on the retrieval results; for short documents with the number of document tokens below the threshold, a traversal block method is used for question and answer, in which the document content is divided into blocks, and the relevance of each block to the user's question is calculated. Only blocks with high relevance are processed and answers are generated; finally, the answers to each block are aggregated and summarized to form the final answer.

2. A word document question answering method based on a large language model according to claim 1, characterized in that: When preprocessing the document, the position of the image content in the original document is recorded, and the original document content and semantic coherence are maintained through special image marking and replacement; the specific implementation is as follows: When a user uploads a Word document containing pictures, a unique number is generated for each picture and saved as an independent file; at the same time, the picture is replaced with a special tag in the original document; in subsequent processing, the description information of the picture is accurately inserted into the tag position, thereby maintaining the content and semantic consistency of the original document.

3. A word document question answering method based on a large language model according to claim 2, characterized in that: The special mark design of the picture is as follows: [PIC_PLACEHOLDER_START:PIC_001] {{PIC_001_DESC}} [PIC_PLACEHOLDER_END:PIC_001].

4. A word document question answering method based on a large language model according to claim 1, 2 or 3, characterized in that: The specific process of document preprocessing is as follows: (1) The user opens a new session and uploads a Word document in the dialogue interface. The system preprocesses the received Word document: first, the document structure is parsed and the document is traversed. When traversing the document, each picture encountered is numbered and saved as an image file, and the picture is replaced with a special mark in the document. In addition, the text content of the previous and next paragraphs of the picture is stored as a file named "picture number.txt" to be used as auxiliary context information when calling the multimodal model to interpret the picture. When a table is encountered, it is converted into a markdown table format to facilitate the subsequent transmission of the data information in the table to the large language model. (2) Use a multimodal large language model to describe and interpret image files. When organizing the prompt words of the multimodal model, the "image number.txt" file will be read. By combining the context of the paragraph where the image is located, rich semantic information is provided to the multimodal model, thereby helping the model to better understand the content and meaning of the image; (3) inserting the description and interpretation information of the image into the special mark reserved for it in the original document according to its number, replacing the placeholder description part in the special mark, so that the document can maintain the original content order; (4) After the above steps, the images and tables in the document passed to this session are converted into text form, and the total token number of the document is calculated. If the number of document tokens exceeds the specified threshold, it is divided into blocks, vectorized, and stored in the vector database, and the variable RAG_ENABLE is set to True; if the number of document tokens does not reach the specified threshold, no processing is performed, and only the variable RAG_ENABLE is set to False.

5. A word document question answering method based on a large language model according to claim 4, characterized in that: The specific implementation process of the dialogue stage is as follows: After document preprocessing is complete, the user engages in a question-and-answer dialogue with the system regarding the document content. The user sends a question, and the system checks the value of the current session variable RAG_ENABLE. If RAG_ENABLE = True, the RAG process is executed, the user's question is vectorized, and the document content related to the user's question is retrieved from the vector database. The document content, together with the user's question, forms a prompt word for the large model, which then processes the question and gives an answer to the user's question. If RAG_ENABLE=False, it means that the document content is short. First, the document content is divided into blocks, and a document question-and-answer dialogue is conducted by traversing each block. Before each block dialogue, the relevance of the current block to the user question is calculated. If the relevance reaches the threshold, the large model dialogue for the block is executed. If the relevance does not reach the threshold, it is skipped. After completing the entire document traversal, the answers to the user questions in each block are summarized and summarized by the large model to form the document's final answer to the user question.

6. A word document question answering method based on a large language model according to claim 1, characterized in that: During document preprocessing, the Python-docx library is used to implement document reading and traversal, image extraction and tag replacement, and table conversion to Markdown format.

7. A word document question answering method based on a large language model according to claim 6, characterized in that: The document reading and traversal: Python-docx loads a document and traverses its contents paragraph by paragraph and element by element. By loading the document object, you can access the paragraphs and table elements in the document. The paragraph is the basic unit of document content, and each paragraph object can be used to extract its text content; the table can be accessed through the row and cell structure, so that users can extract data from the table row by row and cell by cell. The image extraction and tag replacement: When processing images in documents, Python-docx can identify image elements in the document and obtain their location information in the document. It can also extract images and save them as independent files in combination with other tools through the image placeholder information it provides. In addition, Python-docx also supports inserting custom markup content into the document. These tags can be used to subsequently reinsert the description and interpretation of the image into the document, thereby maintaining the integrity and semantic coherence of the document content. The table is converted to Markdown format: Python-docx provides detailed operations on tables in documents. Users can access the rows and cells of the table and extract the data content. In this way, the table data can be converted to other formats.

8. A word document question answering system based on a large language model, characterized in that: include: The image tagging module is used to record the position of the image content in the original document during the document preprocessing stage. After using the multimodal model to describe and interpret the image content, the interpretation result returned by the multimodal model is inserted into the original position of the image, so that the preprocessed content is consistent with the original document sequence, thereby maintaining the original semantic coherence of the text and images in the document; The answer generation module is used to adopt different processing mechanisms based on document length during the dialogue phase: when the number of tokens in a document exceeds the threshold, the document is divided into blocks, vectorized, and stored in the vector database, and the RAG mechanism is enabled. When a user asks a question, the system vectorizes the question and searches the vector database for document content related to the question, generating an answer based on the search results. For short documents with a token count below the threshold, a traversal block approach is used for question and answer, dividing the document content into blocks, calculating the relevance of each block to the user's question, and only processing blocks with high relevance to generate answers. Finally, the answers to each block are aggregated and summarized to form the final answer. The system specifically implements word document question and answer through the method described in any one of claims 1 to 7.

9. A word document question-answering device based on a large language model, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to implement the method according to any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, can implement the method according to any one of claims 1 to 7.

Images