Intelligent question-answering system construction method supporting image enhancement and intelligent question-answering system

Abstract

The invention provides an intelligent question-answering system construction method supporting image enhancement and an intelligent question-answering system, and the method comprises the steps: constructing a user knowledge base, and extracting a picture and a picture title in a document; the extracted picture is stored; searching a title text corresponding to the picture title in the analyzed text segment, and embedding a storage path of the corresponding picture into the title text; representing each text segment as a low-dimensional vector, and storing the low-dimensional vector and the text segments in a vector database; constructing a user question context based on the user knowledge base according to the received user question, and obtaining a text answer of the large language model; and obtaining picture information based on a picture storage path contained in the context of the user question, and displaying the text answer of the large language model and the obtained picture information in a user interaction interface after combining the text answer and the obtained picture information. According to the intelligent question and answer system, the answers not only comprise texts, but also comprise related pictures, so that the user can understand the answer content more intuitively.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to a method for constructing an intelligent question-answering system supporting image enhancement and an intelligent question-answering system. Background Art

[0002] Intelligent question answering is an important subfield of natural language processing. Its core purpose is to develop computer programs that can understand users' natural language questions. The program can not only understand the user's intention, but also give accurate and concise natural language answers based on the user's intention. In recent years, the Large Language Model technology has been continuously iteratively developed. With its powerful text understanding and generation capabilities, as well as logical induction and reasoning capabilities, the Large Language Model has been widely used in text processing tasks such as text classification, intelligent dialogue, and entity recognition.

[0003] Limited by the size of the model training data set, large language models are less effective when processing tasks where relevant data is not included in the training data set. Retrieval Augmented Generation is a technology that uses a user knowledge base to assist large language models in improving their task processing capabilities. In the intelligent question-answering scenario, the retrieval augmented generation technology first retrieves relevant data from the user knowledge base based on the user's question. The relevant data in the user knowledge base will be used as context information for the user's question and input into the large language model together with the user's question. The large language model gives a more accurate and concise answer by understanding and summarizing the input context information.

[0004] The user knowledge base can contain the most recently updated data on the Internet, private business data of financial institutions, confidential documents within enterprises, etc. In order to build a user knowledge base, the documents to be stored are generally parsed and segmented first, and then each text segment is represented as a low-dimensional vector using a text embedding model. This process is called text embedding. Then, each text segment and its corresponding low-dimensional vector are stored in a vector database to facilitate subsequent rapid similarity retrieval.

[0005] However, in the current intelligent question-answering system based on retrieval enhancement generation technology, when parsing document data to build a user knowledge base, only text data is supported, while image data in the document is ignored. In addition, the answers given by the large language model to the user's questions also only contain text. Considering that a considerable number of documents currently contain image information, and image information is often more expressive and easier for users to understand in specific question-answering scenarios involving spatial representation, logical representation, etc., it is of great practical significance to design a method for building an intelligent question-answering system that supports image enhancement and an intelligent question-answering system. Summary of the invention

[0006] In view of the fact that there are often a large number of documents containing image information in the actual intelligent question-answering knowledge base, but the image information in these documents is basically not utilized, the present invention proposes a method for constructing an intelligent question-answering system supporting image enhancement and an intelligent question-answering system. The following technical solutions are specifically adopted: The first aspect of the present invention provides a method for constructing an intelligent question answering system supporting image enhancement, comprising the following steps: Build a user knowledge base, parse and segment the text content of the documents to be stored, and extract the pictures in the documents and the picture titles contained in the pictures; Save the extracted pictures and construct a mapping from picture title to picture storage path; Searching for a title text corresponding to the picture title in the parsed text segment, and embedding the storage path of the corresponding picture into the text segment where the title text is located based on the mapping; Use a text embedding model to represent each text segment as a low-dimensional vector, and save the text segment and its corresponding low-dimensional vector into a vector database; Building a user question context based on the user knowledge base according to the accepted user question, sending it together with the user question to the large language model, and obtaining a text answer from the large language model; The image information is obtained based on the image storage path contained in the user question context, and the text answer of the large language model and the obtained image information are combined and displayed on the user interaction interface.

[0007] Furthermore, the pictures in the extracted document and the picture titles contained in the pictures include: Training an image detection model and an image title extraction model based on the selected documents to be stored and / or stored documents; Use the trained image detection model to extract images from the documents to be stored; Use the trained image title extraction model to extract the image title contained in the extracted image.

[0008] Furthermore, the image detection model and the image title extraction model are trained based on the selected documents to be stored and / or stored documents, including: Select some documents to be stored and / or stored in the database to annotate the image information and title information contained in the image, and build an image detection dataset and an image title extraction dataset; The open source model is trained based on the image detection dataset and the image title extraction dataset to obtain an image detection model and an image title extraction model adapted to the current user knowledge base.

[0009] Furthermore, the method of extracting images from the document to be stored by using the trained image detection model includes: Use the image detection model to scan each page of the incoming document to detect whether there is an image and give a detection confidence level for each detection result; If the confidence is lower than the threshold, the detection result is ignored. Otherwise, the detected image is extracted according to the position of the image border in the page given by the model.

[0010] Furthermore, the extracted images are stored in a local file system or a remote object storage service, and different storage terminals correspond to different image file storage path formats.

[0011] Furthermore, searching for the title text corresponding to the picture title in the parsed text segment, and embedding the storage path of the corresponding picture into the text segment where the title text is located based on the mapping includes: For each detected image, obtain the page number information of the page where the image is located; Based on the page number information, the text segments in the current page and a preset number of adjacent pages corresponding to the page number are detected to find the title text corresponding to the picture title; The image storage path is obtained based on the mapping from the image title to the image storage path, and the image storage path is inserted into the title text in the corresponding text segment.

[0012] Further, constructing a user question context based on the user knowledge base according to the accepted user question includes: Calculate the low-dimensional vector of the user's question through the embedding model; Retrieving top K approximate low-dimensional vectors from the vector database of the user knowledge base using an ANN index; The user question context is constructed through text segments corresponding to K approximate low-dimensional vectors.

[0013] Furthermore, the construction of the user knowledge base also includes: Documents to be stored are classified. For plain text documents, only the text content is parsed and segmented. For documents containing pictures, in addition to text content parsing and segmentation, pictures and picture titles contained in the pictures are also extracted.

[0014] The second aspect of the present invention further provides an intelligent question-answering system supporting image enhancement, which is constructed based on the method described in the first aspect above, and includes: The document parsing module is used to pre-process the documents stored in the user knowledge base, including parsing and segmenting the text content, extracting the pictures and the picture titles contained in the pictures, saving the extracted pictures and constructing a mapping from the picture title to the picture storage path, calling the text embedding model to generate a low-dimensional vector of the text segmentation, and providing a document parsing calling interface for other modules; The vector database module is used to store the core data in the user knowledge base, including the document segments and corresponding low-dimensional vectors generated by the document parsing module; to establish full-text indexes for the text segments and ANN indexes for the low-dimensional vectors; and to provide similar retrieval interfaces for text segments to other modules; The file storage module is used to store the original files of the documents in the user knowledge base and the image files extracted by the document parsing module, and provide a file access interface for other modules; The model management module is used to manage all models that provide image detection, title extraction, text embedding, and conversation chat functions in the entire system, as well as model file storage, model startup initialization, and model usage interface; The question-answering interaction module is used to provide users with a user interaction interface of the intelligent question-answering system.

[0015] Furthermore, the document parsing module is also used to configure parsing strategies to classify documents to be stored in the database. For plain text documents, only the text content is parsed and segmented. For documents containing pictures, in addition to text content parsing and segmenting, pictures and picture titles contained in the pictures are also extracted.

[0016] The beneficial effects of the present invention are as follows: 1) The method for constructing an intelligent question-answering system supporting image enhancement and the intelligent question-answering system proposed in the present invention can parse and extract image information contained in user knowledge base documents, and the answers to terminal users' questions include both text and related images. Compared with traditional text question-answering applications, users can understand the content of the answers more intuitively through the images contained in the answers, which greatly improves the user experience.

[0017] 2) The method for constructing an intelligent question-answering system supporting image enhancement and the intelligent question-answering system proposed in the present invention train image detection models and title extraction models based on documents in a user knowledge base; the trained models can better adapt to the features of images in documents in the user knowledge base, and parse and extract document images more accurately and efficiently.

[0018] 3) The intelligent question-answering system construction method and intelligent question-answering system supporting image enhancement proposed in the present invention provide different document parsing strategies for plain text documents and documents containing images, considering that the parsing and extraction of document images are time-consuming. Users can decide whether to enable image enhancement based on the actual document type.

[0019] 4) The method for constructing an intelligent question-answering system supporting image enhancement and the intelligent question-answering system proposed in the present invention adopt a modular design with decoupling of each module, and are characterized by flexible deployment, convenient upgrading, scalability, and easy maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the method for constructing an intelligent question-answering system supporting image enhancement proposed by the present invention.

[0021] Figure 2 This is an example of the user question and answer result interface in the method for building an intelligent question and answer system supporting image enhancement proposed in the present invention.

[0022] Figure 3 This is a schematic diagram of the architecture of an embodiment of the intelligent question-answering system supporting image enhancement proposed in the present invention. DETAILED DESCRIPTION

[0023] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0024] See also Figure 1 The embodiment of the present invention provides a method for constructing an intelligent question answering system supporting image enhancement, comprising the following steps: S1. Build a user knowledge base, parse and segment the text content of the documents to be stored, and extract the pictures in the documents and the picture titles contained in the pictures.

[0025] As a preferred implementation scheme, in this embodiment, the images in the document and the image titles contained in the images are extracted through a pre-trained image detection model and an image title extraction model. Since the open source target detection model and information extraction model cannot well identify the unique image features and image title features in the user knowledge base documents, the accuracy of image detection and title extraction is low. In order to improve the accuracy of detecting images and extracting titles from user knowledge base documents, it is necessary to use some documents in the knowledge base to construct a training data set to train models with higher accuracy in detecting images and extracting titles. The specific process is as follows: First, select some documents to be stored and / or stored in the database to annotate the image information and the title information contained in the image, and build an image detection dataset and an image title extraction dataset. Specifically, select some documents in the knowledge base that cover most of the image features, and convert each page of this part of the document into a picture containing text and pictures. Then mark the border position of the image part in the picture to build an image detection dataset. Filter out some pictures with poor quality, and then annotate the picture titles contained in the remaining pictures to build an image title extraction dataset.

[0026] The open source model is trained based on the image detection dataset and the image title extraction dataset to obtain the image detection model and image title extraction model adapted to the current user knowledge base. Specifically, based on the open source target detection model such as yolo and detectron, the model is trained using the training dataset constructed in the previous step to obtain the image detection model adapted to the user knowledge base document image extraction task. Based on the open source image information extraction model such as Donut, the model is trained using the image title extraction dataset constructed in the previous step to obtain the image title extraction model adapted to the user knowledge base document.

[0027] Then, use the trained image detection model to scan each page of the incoming document to detect whether there is an image and give a detection confidence for each detection result; If the confidence is lower than the threshold, the detection result is ignored. Otherwise, the detected image is extracted according to the position of the image border in the page given by the model.

[0028] Finally, the trained image title extraction model is used to extract the image title contained in the extracted image.

[0029] S2. Save the extracted pictures and construct a mapping from the picture title to the picture storage path.

[0030] As a preferred implementation scheme, in this embodiment, the extracted pictures are stored in a local file system or a remote object storage service, and different storage terminals correspond to different picture file storage path formats.

[0031] S3. Searching for a title text corresponding to the picture title in the parsed text segment, and embedding the storage path of the corresponding picture into the text segment where the title text is located based on the mapping.

[0032] Consider that for an image in a document, the text that references the image will most likely appear on a page adjacent to the page where the image is located. For example, page 10 of the document contains Figure 1 .2.3, then reference Figure 1 The text fragments of .2.3 appear in most cases between pages 8 and 12 of the document.

[0033] Based on this, as a preferred implementation scheme, in this embodiment, for each detected picture, the page number information of the page where the picture is located is obtained; According to the page number information, only the text segments in the current page corresponding to the page number and a preset number of adjacent pages (such as 3 pages before and after) are detected to find the title text corresponding to the picture title, so as to improve the detection efficiency; Finally, the image storage path is obtained based on the mapping from the image title to the image storage path, and the image storage path is inserted into the title text in the corresponding text segment.

[0034] For example, if the document page 10 contains Figure xx3 and Figure xx4, and the document page 11 contains Figure xx5, the mapping of the constructed picture title to the picture storage path is: Page 10 <Figure xx3: / path / to / xx3.jpg, Figure xx4: / path / to / xx4.jpg>, Page 11 <Figure xx5: / path / to / xx5.jpg>. When detecting text segments, for a text segment on page 9, the content is "…The sample for tissue uniformity inspection is shown in Figure xx4, and its fracture should be perpendicular to…". Since the mapping of the adjacent page 10 contains Figure xx4, and the string "Figure xx4" (i.e., the text title) is also included in the text segment. Therefore, the corresponding picture address is embedded in the text segment, and the text segment content is updated to "…The sample for tissue uniformity inspection is shown in Figure xx4 ( / path / to / xx4.jpg), and its fracture should be perpendicular to…".

[0035] S4. Use the text embedding model to represent each text segment as a low-dimensional vector, and save the text segment and its corresponding low-dimensional vector into the vector database.

[0036] As a preferred implementation scheme, in this embodiment, the vector database establishes an ANN index for low-dimensional vectors to facilitate efficient similarity retrieval, and establishes a full-text index for text segments to facilitate keyword retrieval.

[0037] S5. Build a user question context based on the user knowledge base according to the accepted user question, send it to the large language model together with the user question, and obtain a text answer from the large language model.

[0038] As a preferred implementation, in this embodiment, when conducting question-answering, the low-dimensional vector of the user's question is first calculated through the embedding model; then the top K approximate low-dimensional vectors are retrieved from the vector database of the user knowledge base using the ANN index; and the user's question context is constructed through the text segments corresponding to the K approximate low-dimensional vectors. If the context text segment has referenced the picture title, the corresponding picture address will also be included.

[0039] S6. Obtain image information based on the image storage path contained in the user question context, and combine the text answer of the large language model and the obtained image information to display them on the user interaction interface.

[0040] As a preferred implementation, in this embodiment, the user interaction interface is as follows: Figure 2 As shown in the figure, in response to user questions, not only text answers are given, but also relevant pictures are displayed.

[0041] As a further preferred implementation scheme, in this embodiment, considering that the parsing and extraction operations of document images are relatively time-consuming, when building a user knowledge base, the documents to be stored can also be classified. For plain text documents, only the text content is parsed and segmented. For documents containing images, in addition to text content parsing and segmenting, the images and image titles contained in the images are also extracted, thereby improving the document parsing efficiency.

[0042] See also Figure 3 The second aspect of the present invention further provides an intelligent question-answering system supporting image enhancement, which is constructed based on the method described in the first aspect above, and includes: The document parsing module is used to pre-process the documents stored in the user knowledge base, including parsing and segmenting the text content, extracting the pictures and the picture titles contained in the pictures, saving the extracted pictures and constructing a mapping from the picture title to the picture storage path, calling the text embedding model to generate a low-dimensional vector of the text segmentation, and providing a document parsing calling interface for other modules; The vector database module is used to store the core data in the user knowledge base, including the document segments and corresponding low-dimensional vectors generated by the document parsing module; establish full-text indexes for text segments to facilitate subsequent keyword retrieval, establish ANN indexes for low-dimensional vectors to facilitate subsequent efficient similarity retrieval; and provide similarity retrieval interfaces for text segments to other modules; The file storage module is used to store the original files of the documents in the user knowledge base and the image files extracted by the document parsing module, and provide a file access interface for other modules; The model management module is used to manage all models that provide image detection, title extraction, text embedding, and conversation chat functions in the entire system, as well as model file storage, model startup initialization, and model usage interface; The question-answering interaction module is used to provide users with a user interaction interface of the intelligent question-answering system.

[0043] As a further preferred implementation, in this embodiment, the document parsing module is also used to configure a parsing strategy to classify documents to be stored, and only parse and segment the text content of a plain text document, and extract the image and the image title contained in the image in addition to parsing and segmenting the text content of a document containing an image. Thus, the user can decide whether to enable image enhancement according to the actual document type.

[0044] The following is a further explanation of the interaction between the various modules in the system with reference to specific examples.

[0045] 1) Users upload documents to be stored in the database through the question-and-answer interaction module, and the original documents are saved in the file storage module 2) Users trigger the parsing task of the documents to be stored in the database through the question-answering interaction module: 2.1) The document parsing module first accesses the file storage module to read the original document and start text parsing.

[0046] 2.2) Then access the model management module and use the image detection model and title extraction model managed by it to process the image information in the document.

[0047] 2.3) Finally, save the extracted image to the file storage module and return the image storage address information.

[0048] 2.4) At the same time, save the text segmentation information embedded in the image address to the vector database module.

[0049] 3) When users perform question-answering tasks, they input questions through the question-answering interaction module: 3.1) The question-answer interaction module first retrieves relevant text segments based on the question-to-vector database module and constructs the question context.

[0050] 3.2) Then access the model management module, input the question and related context into the large language model, and get the text response from the model.

[0051] 3.3) At the same time, get the original image from the file storage module according to the image address in the context.

[0052] 3.4) Finally, the text response and related image information of the large model are integrated and the final answer is returned to the user.

[0053] It should be noted that the method of the embodiment of the present invention can be performed by a single device, such as a computer or a server. The method of this embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present invention, and the multiple devices will interact with each other to complete the described method.

[0054] It should be noted that some embodiments of the present invention are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0055] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing an intelligent question-answering system supporting image enhancement, characterized in that: The steps include: Build a user knowledge base, parse and segment the text content of the documents to be stored, and extract the pictures in the documents and the picture titles contained in the pictures; Save the extracted pictures and construct a mapping from picture title to picture storage path; Searching for a title text corresponding to the picture title in the parsed text segment, and embedding the storage path of the corresponding picture into the text segment where the title text is located based on the mapping; Use a text embedding model to represent each text segment as a low-dimensional vector, and save the text segment and its corresponding low-dimensional vector into a vector database; Building a user question context based on the user knowledge base according to the accepted user question, sending it together with the user question to the large language model, and obtaining a text answer from the large language model; The image information is obtained based on the image storage path contained in the user question context, and the text answer of the large language model and the obtained image information are combined and displayed on the user interaction interface.

2. The method for constructing an intelligent question-answering system supporting image enhancement according to claim 1, characterized in that: The pictures in the extracted document and the picture titles contained in the pictures include: Training an image detection model and an image title extraction model based on the selected documents to be stored and / or stored documents; Use the trained image detection model to extract images from the documents to be stored; Use the trained image title extraction model to extract the image title contained in the extracted image.

3. The method for constructing an intelligent question-answering system supporting image enhancement according to claim 2, wherein: The image detection model and image title extraction model based on the selected documents to be stored and / or stored documents include: Select some documents to be stored and / or stored in the database to annotate the image information and title information contained in the image, and build an image detection dataset and an image title extraction dataset; The open source model is trained based on the image detection dataset and the image title extraction dataset to obtain an image detection model and an image title extraction model adapted to the current user knowledge base.

4. The method for constructing an intelligent question-answering system supporting image enhancement according to claim 2, wherein: The method of using the trained image detection model to extract images from the document to be stored includes: Use the image detection model to scan each page of the incoming document to detect whether there is an image and give a detection confidence level for each detection result; If the confidence is lower than the threshold, the detection result is ignored. Otherwise, the detected image is extracted according to the position of the image border in the page given by the model.

5. The method for constructing an intelligent question-answering system supporting image enhancement according to claim 1, wherein: The extracted images are stored in the local file system or remote object storage service. Different storage terminals correspond to different image file storage path formats.

6. The method for constructing an intelligent question-answering system supporting image enhancement according to claim 1, wherein: The step of searching for the title text corresponding to the picture title in the parsed text segment and embedding the storage path of the corresponding picture into the text segment where the title text is located based on the mapping includes: For each detected image, obtain the page number information of the page where the image is located; Based on the page number information, the text segments in the current page and a preset number of adjacent pages corresponding to the page number are detected to search for the title text corresponding to the picture title; The image storage path is obtained based on the mapping from the image title to the image storage path, and the image storage path is inserted into the title text in the corresponding text segment.

7. The method for constructing an intelligent question-answering system supporting image enhancement according to claim 1, wherein: The constructing the user question context based on the user knowledge base according to the accepted user question comprises: Calculate the low-dimensional vector of the user's question through the embedding model; Retrieving top K approximate low-dimensional vectors from the vector database of the user knowledge base using an ANN index; The user question context is constructed through text segments corresponding to K approximate low-dimensional vectors.

8. The method for constructing an intelligent question-answering system supporting image enhancement according to any one of claims 1 to 7, characterized in that: The construction of the user knowledge base also includes: Documents to be stored are classified. For plain text documents, only the text content is parsed and segmented. For documents containing pictures, in addition to text content parsing and segmentation, pictures and picture titles contained in the pictures are also extracted.

9. An intelligent question-answering system supporting image enhancement, constructed based on the method according to any one of claims 1 to 8, characterized in that: include: The document parsing module is used to pre-process the documents stored in the user knowledge base, including parsing and segmenting the text content, extracting the pictures and the picture titles contained in the pictures, saving the extracted pictures and constructing a mapping from the picture title to the picture storage path, calling the text embedding model to generate a low-dimensional vector of the text segmentation, and providing a document parsing calling interface for other modules; The vector database module is used to store the core data in the user knowledge base, including the document segments and corresponding low-dimensional vectors generated by the document parsing module; to establish full-text indexes for the text segments and ANN indexes for the low-dimensional vectors; and to provide similar retrieval interfaces for text segments to other modules; The file storage module is used to store the original files of the documents in the user knowledge base and the image files extracted by the document parsing module, and provide a file access interface for other modules; The model management module is used to manage all models that provide image detection, title extraction, text embedding, and conversation chat functions in the entire system, as well as model file storage, model startup initialization, and model usage interface; The question-answering interaction module is used to provide users with a user interaction interface of the intelligent question-answering system.

10. The intelligent question-answering system supporting image enhancement according to claim 9, characterized in that: The document parsing module is also used to configure parsing strategies to classify documents to be stored. For plain text documents, only the text content is parsed and segmented. For documents containing pictures, in addition to text content parsing and segmenting, the pictures and picture titles contained in the pictures are also extracted.

Images