Intelligent retrieval question-answering system based on large model

Abstract

The invention relates to the technical field of intelligent question answering, and discloses an intelligent retrieval question answering system based on a large model, which is characterized in that a user question module obtains original question content input by a user; the image recognition and extraction module recognizes the document image and extracts an image text; the task type identification module obtains a question and answer type of a user; the instruction template module set constructs an instruction template required by a task; the knowledge retrieval module outputs a plurality of knowledge entries; the question-answer example module outputs a question-answer example context; and the combined prompt module is used for combining the original question content, the knowledge entries, the question and answer example context, the instruction template and the prompt header to form a large language model input prompt, and outputting a final answer by adopting the fine-tuned large language model. According to the method, the composite question-answering capability of layout perception and domain knowledge enhancement is realized through cooperation of multiple modules such as image recognition and extraction and task type recognition, structured prompts are generated, lightweight fine tuning is combined, the cross-domain application cost is reduced, and the method is suitable for multi-scene intelligent question-answering.

Description

Technical Field

[0001] This invention relates to the field of intelligent question answering technology, and in particular to an intelligent retrieval and question answering system based on a large model. Background Technology

[0002] With the development of information technology, intelligent question answering for document images has become a key technology for scenarios such as intelligent digital libraries and enterprise document management. Traditional document image question answering systems are mostly based on multimodal pre-trained models. Although they can integrate text, visual, and layout information, they require targeted fine-tuning with a large amount of data for different tasks, resulting in high training costs and difficulty in applying them to data-scarce scenarios (such as ancient books and historical documents). While large language models, represented by ChatGPT, possess powerful zero-shot learning capabilities and generalization, they can only process plain text instructions and cannot directly understand the layout structure in document images. If text is directly extracted and concatenated through OCR, the structural information contained in two-dimensional layouts such as tables and lists will be lost, leading to a decrease in answer accuracy. In addition, existing technologies lack effective integration of vertical domain knowledge when dealing with domain-specific questions, and large language models are prone to "illusion" problems due to insufficient domain knowledge. How to enable large language models to effectively utilize the layout information of document images and combine domain-specific knowledge to achieve accurate question answering without extensive fine-tuning has become an urgent technical challenge. Summary of the Invention

[0003] This invention provides an intelligent retrieval and question answering system based on a large model. It achieves composite question answering capabilities with layout awareness and domain knowledge enhancement through the collaborative efforts of multiple modules such as image recognition and extraction and task type recognition. It generates structured prompts and combines them with lightweight fine-tuning to reduce the cost of cross-domain applications and is suitable for intelligent question answering in multiple scenarios.

[0004] This invention provides an intelligent retrieval and question-answering system based on a large model, comprising a user question module, a knowledge retrieval module, a question-answer example module, a combined prompt module, an image recognition and extraction module, a task type recognition module, an instruction template module, and a large language model module. The user question module is connected to the knowledge retrieval module, the question-answer example module, the image recognition and extraction module, and the task type recognition module. The image recognition and extraction module is also connected to the knowledge retrieval module. The task type recognition module is also connected to the instruction template module. The knowledge retrieval module, the question-answer example module, and the instruction template module are all connected to the combined prompt module, and the combined prompt module is connected to the large language model module.

[0005] The user question module is used to obtain the original question content input by the user; wherein, the original question content includes question text and document images;

[0006] The image recognition and extraction module is used to use OCR to recognize and extract text segments and corresponding text boxes from the document image, so as to obtain image text with layout information;

[0007] The task type identification module is used to obtain the user's question and answer type based on the question text; wherein, the question and answer type includes open-ended questions and answering questions and answering questions.

[0008] The instruction template module is used to construct instruction templates for task requirements based on question and answer types and three document image question and answer datasets, including DocVQA task requirement instruction templates, InfographicVQA task requirement instruction templates, and MP-DocVQA task requirement instruction templates.

[0009] The knowledge retrieval module is used to calculate and output the multiple knowledge entries with the highest scores based on the question text and image text using a question encoder and a document encoder.

[0010] The question-and-answer example module is used to output question-and-answer example context based on the question text and image text using a set vectorization model;

[0011] The combined prompting module is used to combine the original question content, knowledge entries, question and answer example context, instruction template and prompt header to form a large language model input prompt;

[0012] The large language model module is used to fine-tune the large language model based on the input prompts, and output the final answer using the fine-tuned large language model.

[0013] Furthermore, given a document image and a question-answer pair, the image recognition and extraction module uses an OCR tool to process the document image, extracting text segments and corresponding text boxes. Then, it uses the obtained text box coordinates to estimate the number of spaces and line breaks that need to be filled between text segments in the original image, and fills the gaps based on the estimation results to obtain text with layout information. The specific steps are as follows:

[0014] Rearrange the text segments in order from top to bottom and from left to right based on the two-dimensional coordinates of the text box;

[0015] Group all text segments by line, group text segments and text boxes belonging to the same line into the same group according to coordinates, and calculate the total number of characters and the total width of each line. The total width of each line is equal to the width of the union of all text boxes in the line.

[0016] Estimate the character width in the document image. The character width in the document image is defined as the width of the line with the most characters divided by the number of characters in that line.

[0017] Connect all text segments in the same line from left to right using a set number of spaces. Given two adjacent text segments and their text boxes in each line, the number of spaces required to connect them is equal to the distance between the left edge of the left text box and the right edge of the right text box divided by the character width of the document image.

[0018] By connecting different lines of text from top to bottom using line breaks, we obtain layout-aware text.

[0019] Furthermore, in the instruction template module, the DocVQA task requires the instruction template to be an extractive document image question answering task, containing several document images and questions constructed based on the document images. The answer to each question comes from the text in a document image. The first and second lines of the DocVQA task requires the instruction template to explain the specific requirements of extractive question answering to the model in detail according to the task description provided by the DocVQA dataset; lines 3 to 6 provide placeholders for layout-aware text and questions. When processing each document image, the placeholders in the template will be replaced with the actually constructed layout-aware text and question text; lines 7 to 10 provide the question requirements and confidence requirements.

[0020] Furthermore, in the instruction template module, the InfographicVQA task requirement instruction template contains several infographics. The answer requirements are described in detail in lines 1 to 6. In particular, the content and format of the answers are described in detail in lines 3 to 6, so that the model can fully understand all possible answer formats. Placeholders for layout-aware text and questions are provided in lines 8 and 10. The task requirements are summarized and reiterated in line 12, and the remaining lines are line breaks.

[0021] Furthermore, in the instruction template module, the MP-DocVQA task requires each question in the instruction template to correspond to two document images. The model needs to accurately extract the correct answer from multiple candidate document images. The task is processed using the maximum confidence setting. Under this setting, the model first extracts a candidate answer from each candidate document image and gives the confidence level corresponding to the candidate answer. Then, it selects the candidate answer with the highest confidence level as the final answer.

[0022] Lines 8 and 9 of the MP-DocVQA task requirement instruction template require the model to output both the answer and the confidence score; the rest of the template is similar to the DocVQA task requirement instruction template.

[0023] Furthermore, the knowledge retrieval module employs a question encoder F. q and document encoder F d To retrieve information from a pre-built local knowledge base based on a given question x. K knowledge items were retrieved from the database;

[0024] The knowledge retrieval module encodes questions and knowledge entries into dense feature vectors F, respectively. q (x)∈R h and F d (z)∈R h The scoring function is used to retrieve the knowledge entry for each question, as the inner product between x and z:

[0025]

[0026] The training objective of the knowledge retrieval module is to maximize the score r(x,z) when the knowledge item z is relevant to the answer question x. A pseudo-relevance label is constructed as a training agent, and a pseudo-relevance function H(z,a) is introduced, which has a value of 1 when the knowledge item z contains the keyword in the actual answer a, and 0 otherwise. Simple string matching is used to determine whether the knowledge item z contains the keyword in the actual answer a.

[0027] For each question-answer pair (x, a), extract a positive knowledge entry z for training. + (x), and then use batch negative sampling to extract all samples except z from the batch. + All knowledge entries other than (x) are considered negative for (x,a); negative knowledge entries are denoted as N(x,a), and the scores of positive knowledge entries are denoted as... The loss function LDPR for the knowledge retrieval module is:

[0028]

[0029] Finally, the knowledge retrieval module returns the K highest-scoring knowledge items based on a given question x, denoted as: The retrieval score is defined as:

[0030]

[0031] Where θ represents the model parameters of the knowledge retrieval module.

[0032] Furthermore, the question-answering example module is used to help the large language model better understand the intent of the user's question, given a test input x and any training input x. i The corresponding vectors e and e' are obtained using a vectorized model. i If e and e i Proximities in the embedding space indicate a greater likelihood of sharing similar answers and question inputs; cosine similarity is calculated between the fused features of the test input and each training input, and the top N nearest neighbors in the embedding space are selected as the question-answering example context.

[0033]

[0034] in, It is the set of indices of the top N similar samples in the training set, and the question-answering example context is defined as:

[0035]

[0036] Meanwhile, the training input vectors can be pre-computed and stored to enable efficient question-answering example context selection.

[0037] Furthermore, the combined prompt module uses knowledge entries and question-and-answer example contexts retrieved from the local knowledge base to combine and obtain a prompt P that helps answer the user's question. This prompt P consists of a prompt header h, a set of question-and-answer example contexts ε, a set of related knowledge documents L, a task instruction template V, and a user input x. The overall input prompt for generating the response is as follows:

[0038] P = "[h][ε][L][V][x]"

[0039] The relevant knowledge document L is obtained by extracting relevant data IDs from knowledge entries retrieved by the combined knowledge retrieval module, and then using these IDs to query the online database. The corresponding knowledge document is represented as follows:

[0040] L={d1,d2,…,d n}

[0041] Where n≤K, because there may be the same ID number among the K knowledge items, the question-and-answer example context is obtained from the N question-and-answer example contexts ε={c1,c2,…,c N The study concluded that for the test input, its template is similar to the examples in the context, except that the answer slot is empty, to be filled by the large language model. The generated hints are integrated with the original input into the large language model to improve the accuracy and reliability of the large language model's answers. Finally, the large language model generates the final answer a based on the hint P obtained from question x, i.e., a = f. LLMs (a|P).

[0042] The beneficial effects of this invention are as follows:

[0043] This invention achieves precise capture of document image layout information and intelligent judgment of task type through the collaboration of a user question module, an image recognition and extraction module, and a task type recognition module. Combined with predefined task templates such as DocVQA, InfographicVQA, and MP-DocVQA in the instruction template module, it can specifically guide the large language model to generate answers that meet task requirements, avoiding the answer bias caused by the loss of layout information in traditional methods. The knowledge retrieval module and question-answering example module provide domain knowledge enhancement and few-shot learning support for the large language model through comparative learning and semantic similarity retrieval, effectively solving the illusion problem of large language models in vertical domains. The combined prompt module integrates multi-source information into structured prompts. Combined with the lightweight fine-tuning strategy of the large language model module, it achieves efficient adaptation without a large amount of labeled data, significantly improving zero-shot question-answering performance on datasets such as DocVQA, while reducing the cost of cross-domain applications and improving the efficiency and accuracy of users obtaining information. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the structure of the intelligent retrieval and question answering system based on a large model according to the present invention.

[0045] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0046] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0047] like Figure 1 As shown, this invention provides an intelligent retrieval and question-answering system based on a large model, including a user question module, a knowledge retrieval module, a question-answer example module, a combined prompt module, an image recognition and extraction module, a task type recognition module, an instruction template module, and a large language model module. The user question module is connected to the knowledge retrieval module, the question-answer example module, the image recognition and extraction module, and the task type recognition module. The image recognition and extraction module is also connected to the knowledge retrieval module. The task type recognition module is also connected to the instruction template module. The knowledge retrieval module, the question-answer example module, and the instruction template module are all connected to the combined prompt module. The combined prompt module is connected to the large language model module.

[0048] (1) User Issues Module

[0049] The user question module is used to obtain the original question content input by the user; wherein, the original question content includes question text and document image.

[0050] Document images, which are scanned images of paper documents, are an important information carrier generated after the digitization of documents in digital libraries. Intelligent question answering based on document images requires a full understanding of the content within the images to accurately answer questions, and is one of the key technologies for realizing the intelligence of digital libraries. The three modalities of information contained in document images—text, visual, and layout—are usually complementary, together forming complete document information. Therefore, this paper proposes an intelligent question answering system for document images based on a multimodal pre-trained model to effectively integrate textual, visual, and layout information.

[0051] In this invention, a domain knowledge base is given. Each z i This represents a specific domain knowledge item. Simultaneously, a question-and-answer set is provided. Where x i To represent a specific natural language problem, a i This is the answer to the corresponding question. The question-and-answer set provides specific question-and-answer examples to help the model learn how to use information from the domain knowledge base to answer questions. Let f LLMs This represents a generative large-scale language model, capable of generating coherent natural language text from a given context. In domain-specific question-answering systems, let f... LLMs Given a domain knowledge base Z and a question-answer set D, generate an answer a for each natural language test question x.

[0052] The local knowledge base is primarily used to connect with online databases. Its construction is based on academic documents stored in the online database. Each document is split into different fields through enumeration. Then, each split field is associated with its corresponding unique identifier (ID) in the database. This establishes a close and precise connection between the local knowledge base and the online database. Subsequently, a context-sensitive prompt will be used to retrieve the corresponding document from the database based on the ID. For example, given an academic topic "large model + knowledge graph," the relevant descriptive information and related topics in the documents for that topic are split and then combined with the ID corresponding to that topic.

[0053] The domain-specific question-answering dataset is composed of question-answer pairs. Based on academic documents stored in an online database, each document was split according to different fields, resulting in approximately 100,000 question-answer pairs. The data underwent both automatic and manual filtering. Specifically, data with excessively short answers, most of which did not address any actual content, was automatically filtered out. Data with low-quality answers was also manually filtered out. This domain-specific question-answering dataset will then be used to fine-tune a large language model, with the data randomly shuffled before being input into the model for further tuning.

[0054] (2) Image recognition and extraction module

[0055] The image recognition and extraction module is used to use OCR to recognize and extract text segments and corresponding text boxes from the document image, thereby obtaining image text with layout information.

[0056] By analyzing the text segments and text box coordinates generated by the OCR tool, the relative distances between text segments in the document image are estimated. Then, an appropriate number of spaces and line breaks are added between the text segments to simulate their relative positional relationships in the original document image, forming text with layout information (layout-aware text, i.e., image text). Simultaneously, since different document image question-answering scenarios typically have different requirements for the source and format of answers, different instruction templates are designed for different tasks to ensure the large language model can accurately understand the specific requirements of each task. These text instructions guide the large language model to generate answers that meet the task requirements. The layout-aware text, the question, and the task requirement instruction template together constitute the instruction prompt, which is used as part of the combined prompt to form the input prompt for the large language model to generate the final answer.

[0057] Given a document image and a question-answer pair, use an OCR tool to process the document image, extracting text segments and corresponding text boxes. Then, use the obtained text box coordinates to estimate the number of spaces and line breaks that need to be filled between text segments in the original image, and fill them in according to the estimation results to obtain text with layout information. The specific steps are as follows:

[0058] 1) Because the order of text segments obtained by the OCR tool is not exactly the same as the order in which they appear in the original document, it is necessary to rearrange the text segments in order from top to bottom and from left to right according to the two-dimensional coordinates of the text box.

[0059] 2) Group all text segments by line. Group text segments and text boxes belonging to the same line into the same group based on coordinates, and calculate the total number of characters and the total width of each line. The total width of each line is equal to the width of the union of all text boxes in the line (the horizontal distance between the left edge of the leftmost text box and the right edge of the rightmost text box).

[0060] 3) Estimate the character width in the document image. The character width in the document image is defined as the width of the line with the most characters divided by the number of characters in that line.

[0061] 4) Connect all text segments in the same line from left to right with an appropriate number of spaces. Given two adjacent text segments and their text boxes in each line, the number of spaces required to connect them is equal to the distance between the left edge of the left text box and the right edge of the right text box divided by the character width of the document image (rounded down).

[0062] 5) Connect different lines of text from top to bottom using newline characters to obtain layout-aware text. For convenience, a single newline character is used by default to connect two lines of text, and exploration of more line-connecting strategies will be left for future work.

[0063] (3) Task type identification module

[0064] The task type identification module is used to obtain the user's question and answer type based on the question text; wherein, the question and answer type includes open-ended questions and answering questions and answering questions.

[0065] Different document image question answering scenarios typically have different requirements for the form of the answers. For example, open-ended question answering scenarios usually require the model to provide answers beyond the information in the document image itself, based on its own knowledge. Extractive question answering, on the other hand, requires the model to extract the answer to the question directly from the given document image. The classic document image question answering evaluation task DocVQA belongs to extractive question answering, which requires the model to extract the answer to the question directly from the document image. If the question and layout-aware text are directly used as input to the model without restrictions, the model can easily generate open-ended answers that do not meet the task requirements. For example, the generated answer may not be in the original document image or may generate unnecessary explanations and descriptions.

[0066] (4) Instruction Template Module

[0067] To ensure the proposed method can be effectively applied to various document and image-based intelligent question-answering scenarios, task requirements are incorporated into the model's instruction text in the form of textual descriptions by constructing task requirement instruction templates. This allows the model to generate answers that meet the task requirements. Specifically, a unique prompt template with a detailed task requirement description is designed for each task, and each template includes layout-aware text and placeholders for the question.

[0068] Different prompt templates were designed for three document image question-answering datasets: DocVQA, InfographicVQA, and MPDocVQA.

[0069] 1) The DocVQA dataset is an extractive document image question answering task, containing 12,767 document images and 50,000 questions constructed from document images. The answer to each question comes from the text in a document image. Lines 1 and 2 of the DocVQA task requirement instruction template detail the specific requirements for extractive question answering based on the task description provided with the DocVQA dataset. Lines 3-6 provide placeholders for layout-aware text and questions; these placeholders are replaced with the actual constructed layout-aware text and question text when processing each document image. Lines 7-10 provide the question requirements and confidence requirements. Since the layout-aware text converted from document images is usually quite long, to prevent the model from forgetting the task requirements due to text content interference, line 8 of the template briefly summarizes and reiterates the task requirements. The DocVQA task requirement instruction template is shown in Table 1 below.

[0070] Table 1 DocVQA Task Requirements Instruction Template

[0071]

[0072]

[0073] 2) The InfographicVQA dataset contains 5485 infographics. Compared to DocVQA, the document images in InfographicVQA contain richer visual elements and higher information density. The template details the answer requirements in lines 1-6. Compared to the DocVQA task, the answers in the InfographicVQA task are more complex, categorized into four types. Therefore, the template provides detailed descriptions of the content and format of the answers in lines 3-6, enabling the model to fully understand all possible answer formats. Specifically, line 3 emphasizes that the answer to the question may be a continuous text in the document; line 4 emphasizes that the answer to the question may be composed of multiple continuous text segments in the document; line 5 emphasizes that the answer to the question may be a continuous text in the question itself; and line 6 emphasizes that the answer to the question may be a number not present in the document, and provides examples of situations that might produce such answers. The template provides layout-aware placeholders for the text and the question in lines 8 and 10. Similar to the DocVQA template, to prevent the model from being distracted by text content and forgetting its task, the InfographicVQA template summarizes and reiterates the task requirements in line 12, with the remaining lines containing line breaks. The instruction template for the InfographicVQA task is shown in Table 2.

[0074] Table 2 InfographicVQA Task Requirements Instruction Template

[0075]

[0076]

[0077] 3) The MP-DocVQA dataset extends the DocVQA dataset to a more realistic multi-document image question-answering scenario. Each question in this dataset corresponds to 20 document images, and the model needs to accurately extract the correct answer from multiple candidate document images. A maximum confidence setting is used to handle this task. Under this setting, the model first extracts a candidate answer for each candidate document image and provides the corresponding confidence score, then selects the candidate answer with the highest confidence score as the final answer. Lines 8 and 9 of the prompt template require the model to output both the answer and the confidence score; the rest is similar to the DocVQA prompt template. The MP-DocVQA task requirement instruction template is shown in Table 3.

[0078] Table 3 MP-DocVQA Task Requirements Instruction Template

[0079]

[0080]

[0081] The implementation of instruction prompting relies primarily on the ability of a large language model to understand layout through spaces and newlines. This invention proposes an instruction fine-tuning method (LATIN-Tuning), the core of which is to construct a set of instruction fine-tuning samples. The construction process of the instruction fine-tuning data is as follows: First, starting from CSV format tabular data, a set of layout-aware text filled with spaces and newlines is constructed using the Pandas package. Specifically, the original CSV tabular data is first read using the Pandas package, converted to DataFrame format, and then output as plain text, retaining the automatically filled spaces and newlines. Then, using prompt learning, a question-answer pair is constructed for each layout-aware text using a large model. The layout-aware text is input into the Claude model, and the model is prompted to construct a question-answer pair, requiring the generated question to involve understanding the text layout. Finally, the Alpaca model is fine-tuned on the obtained instruction fine-tuning dataset to enhance the model's ability to understand layout through spaces and newlines. Specifically, the obtained layout-aware text and the generated question are concatenated as the input to the fine-tuning dataset, and the generated answer is used as the output of the fine-tuning dataset.

[0082] (5) Knowledge Retrieval Module

[0083] The knowledge retrieval module is used to calculate and output the multiple knowledge entries with the highest scores based on the question text and image text using a question encoder and a document encoder.

[0084] This invention constructs a question-and-answer dataset in the academic domain and uses this dataset to fine-tune a large language model using LoRa. Simultaneously, to accurately retrieve domain knowledge context, a domain knowledge context retriever, i.e., a knowledge retrieval module, is constructed, which is weakly supervised and trained using contrastive learning. Furthermore, question-and-answer example contexts are obtained from the question-and-answer dataset for few-shot learning in LLMs. Finally, a context prompter is designed to combine the aforementioned instruction template, domain knowledge context, and question-and-answer example contexts as prompts, which are then input into the LLMs to generate answers.

[0085] The domain knowledge context retrieval system employs a question encoder F. q and document encoder F d Both are Transformer-like encoders that extract information from a pre-built local knowledge base based on a given question x. K knowledge items are retrieved from the domain; this knowledge is expected to provide useful information for answering the question. The domain knowledge context encodes the question and knowledge items into dense feature vectors F, respectively. q (x)∈R h and F d (z)∈R h The scoring function is used to retrieve the knowledge entry for each question, as the inner product between x and z:

[0086]

[0087] The training objective of the domain knowledge context retrieval tool is to maximize the score r(x,z) when the knowledge item z is relevant to the answer question x. However, since the true relevance between question x and knowledge item z is not easy to obtain, pseudo-relevance labels are constructed as training proxies, and a pseudo-relevance function H(z,a) is introduced. Its value is 1 when the knowledge item z contains the keyword in the actual answer a, and 0 otherwise. Simple string matching is used to determine whether the knowledge item z contains the keyword in the actual answer s.

[0088] For each question-answer pair (x, a), extract a positive knowledge entry z for training. + (x), and then use batch negative sampling to extract all samples except z from the batch. + All knowledge entries other than (x) are considered negative for (x,a); negative knowledge entries are denoted as N(x,a), and the scores of positive knowledge entries are denoted as... The loss function LDPR for the knowledge retrieval module is:

[0089]

[0090] Finally, the knowledge retrieval module returns the K highest-scoring knowledge items based on a given question x, denoted as: The retrieval score is defined as:

[0091]

[0092] Where θ represents the model parameters of the document retrieval system.

[0093] (6) Question and Answer Example Module

[0094] The question-and-answer example module is used to output question-and-answer example context based on the question text and image text using a set vectorization model.

[0095] The purpose of question-and-answer example context is to help large language models better understand the intent of user questions, given a test input x and any training input x. i The corresponding vectors e and e' are obtained using a vectorization model (such as text2vec-largechinese). i If e and e i Proximities in the embedding space indicate a greater likelihood of sharing similar answers and question inputs; cosine similarity is calculated between the fused features of the test input and each training input, and the top N nearest neighbors in the embedding space are selected as the question-answering example context.

[0096]

[0097] in, It is the set of indices of the top N similar samples in the training set, and the question-answering example context is defined as:

[0098]

[0099] Meanwhile, the training input vectors can be pre-computed and stored to enable efficient question-answering example context selection.

[0100] (7) Combined prompt module

[0101] The combined prompt module is used to combine the original question content, knowledge entries, question and answer example context, instruction template and prompt header to form input prompts for a large language model.

[0102] Large language models have achieved remarkable performance in conversational tasks. However, their output can be unreliable and deceptive. Therefore, this paper proposes a contextual prompter that combines knowledge entries retrieved from a local knowledge base, question-and-answer example contexts, and corresponding instruction templates to provide a helpful suggestion for answering the user's question.

[0103] A prompt P consists of a prompt header h, a set of question-and-answer example contexts ε, a set of related knowledge documents L, a task instruction template V, and a user input x. The overall input prompt for generating a response is as follows:

[0104] P = "[h][ε][L][V][x]"

[0105] The prompt header describes the question-answering task in natural language. While encouraging large language models to generate answers based on retrieved relevant knowledge, it also allows for extensive exploration, generating answers beyond the prompt information. The relevant knowledge document L is obtained by extracting relevant data IDs from individual knowledge entries retrieved by the combined knowledge retrieval module, and then using these IDs to query the online database. The corresponding knowledge document is represented as follows:

[0106] L={d1,d2,…,d n}

[0107] Where n≤K, because there may be the same ID number among the K knowledge items, the question-and-answer example context is obtained from the N question-and-answer example contexts ε={c1,c2,…,c N The study concluded that for the test input, its template is similar to the examples in the context, except that the answer slot is empty, to be filled by the large language model. The generated hints are integrated with the original input into the large language model to improve the accuracy and reliability of the large language model's answers. Finally, the large language model generates the final answer a based on the hint P obtained from question x, i.e., a = f. LLMs (a|P).

[0108] (8) Large Language Model Module

[0109] The large language model module is used to fine-tune the large language model based on the input prompts, and output the final answer using the fine-tuned large language model.

[0110] This invention achieves precise capture of document image layout information and intelligent judgment of task type through the collaboration of a user question module, an image recognition and extraction module, and a task type recognition module. Combined with predefined task templates such as DocVQA, InfographicVQA, and MP-DocVQA in the instruction template module, it can specifically guide the large language model to generate answers that meet task requirements, avoiding the answer bias caused by the loss of layout information in traditional methods. The knowledge retrieval module and question-answering example module provide domain knowledge enhancement and few-shot learning support for the large language model through comparative learning and semantic similarity retrieval, effectively solving the illusion problem of large language models in vertical domains. The combined prompt module integrates multi-source information into structured prompts. Combined with the lightweight fine-tuning strategy of the large language model module, it achieves efficient adaptation without a large amount of labeled data, significantly improving zero-shot question-answering performance on datasets such as DocVQA, while reducing the cost of cross-domain applications and improving the efficiency and accuracy of users obtaining information.

[0111] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0112] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A large model-based intelligent retrieval question and answer system, characterized in that, The user question module is connected with the knowledge retrieval module, the question and answer example module, the image recognition extraction module and the task type identification module, the image recognition extraction module is also connected with the knowledge retrieval module, the task type identification module is also connected with the instruction template module, the knowledge retrieval module, the question and answer example module and the instruction template module are connected with the combination prompt module, and the combination prompt module is connected with the large language model module. The user question module is used for obtaining original question content input by a user; wherein the original question content includes question text and a document image; The image recognition extraction module is used for recognizing and extracting a text segment and a corresponding text box from the document image by using OCR to obtain image text with layout information; The task type identification module is used for obtaining a question and answer type of the user according to the question text; wherein the question and answer type includes open question and answer and extraction question and answer; The instruction template module is used for constructing an instruction template required by a task according to the question and answer type and three kinds of document image question and answer data sets, including a DocVQA task requirement instruction template, an InfographicVQA task requirement instruction template and an MP-DocVQA task requirement instruction template; The knowledge retrieval module is used for calculating a plurality of knowledge entries with the highest output score by using a question encoder and a document encoder according to the question text and the image text; The question and answer example module is used for outputting a question and answer example context by using a set vectorization model according to the question text and the image text; The combination prompt module is used for combining the original question content, the knowledge entry, the question and answer example context, the instruction template and a prompt head to form a large language model input prompt; The large language model module is used for fine-tuning a large language model according to the large language model input prompt, and outputting a final answer by using the fine-tuned large language model.

2. The large model-based intelligent retrieval question answering system according to claim 1, wherein, The image recognition extraction module is used for processing a document image by using an OCR tool to extract a text segment and a corresponding text box in the document image for a given document image and a question and answer pair; then the number of spaces and line breaks required to fill in the blank between the text segments in the original image is estimated by using the obtained text box coordinates, and the blank is filled according to the estimation result to obtain text with layout information, and the specific steps are as follows: The text segments are rearranged according to the two-dimensional coordinates of the text boxes in the order from top to bottom and from left to right; All the text segments are grouped by lines, and the text segments and text boxes belonging to the same line are grouped into the same group according to the coordinates, and the total number of characters of each line and the total width of each line are calculated, wherein the total width of each line is equal to the width of the union of all text boxes in the line; The character width in the document image is estimated, which is defined as the width of the line with the most characters divided by the number of characters in the line; The number of spaces required to connect the two adjacent text segments and their text boxes in each line from left to right is equal to the distance between the left boundary of the left text box and the right boundary of the right text box divided by the character width of the document image; The texts of different lines are connected in turn from top to bottom by using the line break to obtain the layout-aware text. 3.The large model-based intelligent retrieval question answering system according to claim 1, characterized in that, In the instruction template module, the DocVQA task requirement instruction template is used as an extractive document image question answering task, contains a plurality of document images and questions constructed based on the document images, and the answer of each question comes from the text in a document image. The first and second lines of the DocVQA task requirement instruction template specify the specific requirements of the extractive question answering to the model according to the task description provided by the DocVQA data set; the third to sixth lines provide placeholders for layout-aware text and questions, which are replaced by the actual layout-aware text and question text constructed in the template when processing each document image; the seventh to tenth lines provide question requirements and confidence requirements. 4.The large model-based intelligent retrieval question answering system according to claim 1, characterized in that, In the instruction template module, the InfographicVQA task requirement instruction template contains a plurality of information charts, and the first to sixth lines describe the answer requirements in detail. In the third to sixth lines, the content and format of the answer are described in detail, so that the model can fully understand all possible answer forms. The placeholders for layout-aware text and questions are provided in the eighth and tenth lines, and the task requirements are summarized and reiterated in the twelfth line, and the rest are line breaks. 5.The large model-based intelligent retrieval question answering system according to claim 3, characterized in that, In the instruction template module, each question in the MP-DocVQA task requirement instruction template corresponds to two document images, and the model needs to accurately extract the correct answer from multiple candidate document images. The maximum confidence setting is used to process this task. Under this setting, the model first extracts a candidate answer on each candidate document image and gives the confidence of the candidate answer, and then selects the candidate answer with the highest confidence as the final answer. The eighth to ninth lines of the MP-DocVQA task requirement instruction template require the model to output the answer and the confidence at the same time, and the rest are similar to the DocVQA task requirement instruction template. 6.The large model-based intelligent retrieval question answering system according to claim 2, characterized in that, The knowledge retrieval module employs a question encoder F q and a document encoder F d to retrieve K knowledge items from a pre-built local knowledge base according to a given question x; The knowledge retrieval module encodes the questions and knowledge items into dense feature vectors F q (x) ∈ R h and F d (z) ∈ R h The scoring function is used to retrieve the knowledge item for each question as the inner product between the representations of x and z: The training target of the knowledge retrieval module is to maximize the score r(x,z) when the knowledge item z is related to the question x. The pseudo-relevant label is constructed as a proxy for training, and the pseudo-relevant function H(z,a) is introduced, which is 1 when the knowledge item z contains the keywords in the actual answer a, and 0 otherwise. Whether the knowledge item z contains the keywords in the actual answer a is determined by simple string matching; For each question-answer pair (x, a), extract a positive knowledge item z for training + (x), then use batch negative sampling, considering all knowledge items in the batch except z + (x) as negative for (x, a); denote the negative knowledge items as N(x, a) and the score of positive knowledge items as The loss function LDPR of the knowledge retrieval module is: The final knowledge retrieval module returns the top K knowledge items with the highest scores according to the given question x, denoted as The retrieval score is defined as: Where θ represents the model parameters of the knowledge retrieval module.

7. The large model-based intelligent retrieval question answering system according to claim 6, characterized in that, The question-answer example module is used to help large language models better understand the intent of user questions, given a test input x and any training input x i , a vectorization model is used to obtain its corresponding vector e and e i , if the distance between e and e i in the embedding space is close, it is more likely to share similar answers and question inputs; calculate the cosine similarity of the fusion features between the test input and each training input, and select the top N nearest neighbors in the embedding space as the question-answer example context: wherein, is the index set of the top N similar samples in the training set, and the question and answer example context is defined as: At the same time, the vector of the training input can be calculated and stored in advance to realize efficient context selection of the question and answer examples. 8.The large model-based intelligent retrieval question answering system according to claim 7, characterized in that, The combination prompt module adopts the knowledge entries retrieved from the local knowledge base and the question and answer example context to obtain a prompt P that is helpful for answering the user's question by combination, which is composed of a prompt head h, a set of question and answer example contexts ε, a set of related knowledge documents L, a task instruction template V and a user input x, and the overall input prompt of the response is as follows: P = "[h][ε][L][V][x]" The related knowledge documents L are the relevant data id numbers extracted from the knowledge entries retrieved by the combination knowledge retrieval module, and the corresponding knowledge documents are obtained by querying the online database using these id numbers, and the corresponding knowledge documents are represented as: L = {d1, d2,..., d n} wherein n≤K, because there can be the same id number in K knowledge entries, the question and answer example context is derived from the obtained N question and answer example contexts ε={c1, c2, …, c N} for the test input, the template is similar to the example in the context, only the answer slot is blank, for the large language model to fill in; the generated prompt is integrated into the large language model together with the original input to improve the accuracy and reliability of the answer of the large language model, and finally the large language model generates the final answer a according to the question x and the prompt P, that is, a=f LLMs (a|P).

Images