Enterprise document query multi-mode question-answering system generated by combining retrieval enhancement
By combining search-enhanced generation and multimodal technology enterprise document query system, the difficulty of information query caused by enterprise document diversity is solved, and a fast and accurate multimodal question-and-answer question-and-answer is achieved, which significantly improves the efficiency and accuracy of information query.
Patent Information
- Application Number
- CN202411066559.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-22
AI Technical Summary
There are various types of corporate documents, including text, images, voice, etc., which leads to difficulties in information processing and querying. Traditional manual query methods are inefficient and costly, and cannot meet the needs of modern enterprises for fast and accurate information query.
Using a multimodal question and answer system for enterprise document query combining search enhancement generation (RAG) and multimodal technology, text and pictures are vectorized through Sentence-BERT and VGG16 models, and FAISS vector library is used to match, and comparison learning is used. The method draws text and pictures with similar semantics closer to the distance in vector space and generates answers in combination with the big model.
It improves the efficiency and accuracy of enterprise document information query, significantly reduces labor costs, provides fast and accurate multimodal question-and-answer capabilities, and improves search accuracy and answer quality.
Smart Images

Figure CN119003724A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information technology, and in particular to an enterprise document query multimodal question-answering system combined with retrieval enhancement generation. Background Art
[0002] There is a diversity of data in corporate documents, such as corporate rules and regulations, notices, and internal documents. These data contain not only text information, but also images, voice, and other non-text data. However, in actual applications, for a specific question, such as querying the details of a product or the key data of a market research report, it is usually done manually to find relevant documents or rely on the experience of staff to give answers. This manual processing method is not only inefficient, but also has high labor costs, and cannot meet the needs of modern enterprises for fast and accurate information query.
[0003] To address the inefficiency of enterprise document data acquisition, a question-answering system can be used to reduce the workload of manual use of search engines or searching for information, thereby improving efficiency and reducing costs for enterprises. Existing question-answering solutions include question-answering systems based on information retrieval. This method retrieves documents related to user questions from a large document library, which usually uses search engine technology such as BM25 (refer to Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and in Information Retrieval 3,4(2009),333–389.) or TF-IDF (refer to Karen Sparck Jones.1972.Astatistical interpretation of term specificity and its application inretrieval.Journal of documentation 28,1(1972),11–21.), select the most relevant paragraphs or sentences from the retrieved documents, and finally use a ranking algorithm or machine learning model to extract possible answers from the selected paragraphs. This type of method can handle open domain problems and has strong adaptability, but the answer speed depends on the retrieval efficiency, and the answer may not be accurate or specific enough. In addition, there is a question-answering system based on retrieval regeneration, which is implemented by retrieving relevant documents from the document library, inputting the retrieved documents together with the questions into the generation model, and generating specific answers. This solution combines the advantages of retrieval and generation to provide more accurate and context-related answers.
[0004] However, these methods do not support multimodal question answering and cannot process non-text data input such as images and documents, which limits their application in multimodal scenarios. Summary of the invention
[0005] The technical problem to be solved by the present invention is that the data types of enterprise documents are diverse, including text, images, voice and other forms, specifically including but not limited to enterprise rules and regulations, publicity requirements and product promotion planning, various interactive voice records, etc. The diversity and complexity of these data forms make information processing and query extremely difficult, and require a lot of time and resources. The traditional manual query method is inefficient and costly, and it is difficult to meet the needs of modern enterprises for fast and accurate information query.
[0006] In order to solve the above technical problems, the purpose of the present invention is to propose a multimodal question-answering system for enterprise document query combined with retrieval enhancement generation, aiming to improve the efficiency of enterprise document data query.
[0007] In order to achieve the above object, the technical solution adopted by the present invention is:
[0008] A multimodal question-answering system for enterprise document query combined with retrieval-augmented generation is suitable for text and image queries. It is constructed by combining retrieval-augmented generation (RAG) with multimodal technology. An algorithm is used to vectorize existing text and images and store them in a vector library. Then, the distance between text and images with the same semantics in the vector space is shortened through comparative learning. Then, the text and images in the user's question are vectorized and matched in the vector library. Finally, the matching result is concatenated with the question and put into a prompt to be passed into a large model to obtain the answer, thereby realizing multimodal question-answering in enterprise documents.
[0009] As a preferred technical solution of the present invention, the enterprise document query multimodal question answering system implements the following steps:
[0010] First, the Sentence-BERT model is used to vectorize the text to capture the semantic relationship between words and generate high-quality text vectors. At the same time, the VGG16 model is selected to vectorize the image to extract image features.
[0011] Next, the vectors of these texts and images are stored using the FAISS vector library, which has efficient vector retrieval capabilities and can quickly find and match high-dimensional vectors.
[0012] Then, by comparing texts and images with the same semantics, the distance between them in the vector space is shortened to achieve better cross-modal understanding and representation learning; the user's questions will be matched in the vector library through the cosine similarity algorithm to find the most relevant question-answer pairs; during the matching process, the Top-K most similar question-answer pairs are returned to ensure that diverse and relevant answers are provided;
[0013] Finally, the user's question and the matched question-answer pair are embedded in the designed prompt word template and passed to the big model for analysis and answering. In this way, the big model can comprehensively consider the user's question and related context to generate accurate and detailed answers.
[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0015] 1. The present invention proposes a question-answering model that combines RAG and multimodal technology, aiming to improve the efficiency and accuracy of information query of enterprise documents. A dialogue model is proposed for enterprise document data, which can quickly and accurately answer questions raised by users. A contrastive learning method is used to map texts and pictures with similar semantics into a unified vector space, effectively narrowing the semantic gap between different modalities. Experiments have shown that the multimodal question-answering system proposed in the present invention can make accurate and efficient answers to the multimodal data of enterprise documents, effectively reducing labor costs and improving the efficiency of enterprise information query.
[0016] 2. By converting text and image data into high-dimensional vectors and fusing them, the system can provide comprehensive and efficient answers. Experimental results show that the model performs well in terms of retrieval accuracy, answer quality and response time, significantly outperforming previous question-answering systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a framework diagram for enterprise document data management and analysis optimization.
[0018] Figure 2 This is the algorithm flow chart. DETAILED DESCRIPTION
[0019] Introduction to the Retrieval-Enhanced Generation Model (RAG):
[0020] Retrieval-Augmented Generation (RAG) technology (see Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim etal. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–9474.; and Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652 (2023).). Combines the capabilities of retrieval and generation, aiming to improve the performance of question-answering systems using the latest natural language processing techniques. By combining retrieval and generation, RAG provides a new direction for improving the performance and adaptability of large language models in various tasks.
[0021] The RAG dialogue model mainly consists of two key steps: document retrieval and answer generation. First, RAG uses a dense vector retriever to retrieve the most relevant documents or document fragments from a large collection of documents for the user's input question. This step is achieved by encoding the question and all possible documents into vectors in a high-dimensional space and calculating the similarity between them. Then, RAG uses the retrieved relevant documents as context information to generate the final answer to the question through a pre-trained language generation model. This process combines the retrieved information with the model's generation ability to produce more accurate and richer responses.
[0022] The most core components of the RAG dialogue model are the vector database and similarity detection, respectively:
[0023] (1) Vector database
[0024] One of the key components in the RAG model is the vector library. The vector library is responsible for storing and managing vector representations of a large number of documents or document fragments. These vector representations are converted from raw text data by pre-trained encoders, and they contain rich semantic information. In the document retrieval stage of RAG, the vector library plays a key role because it directly affects the efficiency and accuracy of retrieval. Specifically, the vectorization process first requires preprocessing of the raw text, and then converting it into high-dimensional vectors using a pre-trained encoder. Finally, these vectors are stored in an efficient data structure to support fast similarity retrieval.
[0025] (2) Similarity Detection
[0026] Based on the vector representation in the vector library, the RAG model can perform similarity detection to find the document content that is most relevant to the user input. Similarity detection is one of the key steps of the RAG model. It retrieves the most matching document or document fragment by calculating the similarity between the user input and the vector stored in the vector library. Commonly used similarity calculation methods include cosine similarity, Euclidean distance, and inner product. In the similarity retrieval process, the user input is first converted into a vector representation, and then a number of candidate vectors are retrieved from the vector library using an efficient similarity calculation method. Finally, these candidate vectors are sorted and screened to select the most relevant content in the end, providing a basis for the next step of answer generation. The accuracy and speed of similarity detection directly affect the overall performance of the RAG model.
[0027] In response to the limitations of existing dialogue models in multimodal question-answering scenarios, the present invention proposes a solution that combines RAG and multimodal technology: using algorithms to vectorize existing text and images and store them in a vector library, and then using comparative learning to shorten the distance between text and images with the same semantics in the vector space; then vectorize the text and images in the user's question and match them in the vector library; finally, concatenate the matching results with the question, put them into a prompt and pass them into the large model to obtain the answer, thereby realizing multimodal question-answering in corporate documents. Although corporate documents have various data forms, in the present invention's research, the multimodal data of interest is limited to text and images. Please refer to Figure 1 As shown, the present invention proposes a solution combining RAG and multimodal technology to address various limitations of existing enterprise document data analysis.
[0028] The following content illustrates the enterprise document query multimodal question-answering system and simulation experiment proposed by the present invention through Examples 1 and 2 respectively.
[0029] Example 1 Enterprise Document Query Multimodal Question Answering System
[0030] The present invention aims to realize that a large model can accurately answer user needs in enterprise documents. To this end, a variety of technologies and algorithms are used to optimize the processing and matching of text and image data.
[0031] First, the Sentence-BERT model (see Reimers N, Gurevych I. Sentence-bert: Sentence embeddings using siamese bert-networks [J]. arXiv preprint arXiv: 1908.10084, 2019.) is used to vectorize the text to capture the semantic relationship between words and generate high-quality text vectors. At the same time, the VGG16 model (see Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition [J]. arXiv preprint arXiv: 1409.1556, 2014.) is selected to vectorize the image to extract image features.
[0032] Next, the FAISS vector library (see Douze M, Guzhva A, Deng C, et al. The faiss library [J]. arXiv preprint arXiv: 2401.08281, 2024.) is used to store the vectors of these texts and images. The library has efficient vector retrieval capabilities and can quickly search and match high-dimensional vectors.
[0033] Then, by comparing text and images with the same semantics, the distance between them in the vector space can be shortened, achieving better cross-modal understanding and representation learning. The user's question will be matched in the vector library through the cosine similarity algorithm to find the most relevant question-answer pairs. During the matching process, the Top-K most similar question-answer pairs are returned to ensure that diverse and relevant answers are provided.
[0034] Finally, the user's question and the matched question-answer pair are embedded in the designed prompt word template and passed to the big model for analysis and answering. In this way, the big model can comprehensively consider the user's question and related context to generate accurate and detailed answers. Figure 2 The whole algorithm flow is shown.
[0035] The following are detailed descriptions:
[0036] 1. Construction of multimodal vector library
[0037] 1.1 Text Vectorization
[0038] Text vectorization is the process of converting text data into numerical vectors, so that machine learning algorithms can process and understand the text content. The present invention adopts the Sentence-BERT model to generate high-quality word vectors.
[0039] (1) Principle
[0040] The core idea of Sentence-BERT is to combine the powerful pre-training capabilities of BERT with the need for sentence-level embeddings. Unlike traditional BERT, Sentence-BERT is able to directly generate fixed-length sentence vectors, which makes sentence similarity calculations more efficient. Sentence-BERT uses the BERT model to generate contextual embeddings for sentences, and then aggregates these embeddings into a fixed-length sentence vector in some way (such as average pooling). Once the fixed-length vectors for the sentences are generated, Sentence-BERT can calculate the similarity between sentences through simple vector operations (such as cosine similarity).
[0041] For an input sentence S, Sentence-BERT uses the BERT model to generate the context embedding of the sentence. BERT generates a context vector representation for each word:
[0042] E=BERT(S) (3-1)
[0043] Where E is a matrix containing the vector representation of each word in the sentence.
[0044] In order to generate a fixed-length sentence vector, Sentence-BERT usually uses the average pooling method:
[0045]
[0046] where v S is the vector representation of sentence S, n is the number of words in the sentence, E i is the vector representation of the i-th word.
[0047] Once the fixed-length vector v of the sentence is generated S , the similarity between two sentences can be calculated by cosine similarity. For the vector representation v of two sentences S1 and S2 S1 and v S2 :
[0048]
[0049] (2) Workflow
[0050] The workflow of Sentence-BERT includes the following steps:
[0051] (a) Input preprocessing: The user inputs one or more sentences, and Sentence-BERT encodes the input sentences using the BERT model. Each sentence is split into words and converted into word vectors. BERT generates a context vector (embedding) for each word.
[0052] (b) Sentence pooling: By performing a pooling operation (usually average pooling) on the word vectors generated by BERT, the word vectors of each sentence are aggregated into a sentence vector of fixed length.
[0053] (c) Similarity calculation: The generated sentence vectors are used for various downstream tasks, such as calculating the cosine similarity between sentence pairs.
[0054] Sentence-BERT provides an efficient and accurate solution for sentence-level text representation by optimizing and expanding the BERT model. This makes Sentence-BERT valuable in tasks such as question-answering systems, semantic similarity calculation, and text classification.
[0055] 1.2 Image Vectorization
[0056] Image vectorization uses the VGG16 model to vectorize images. VGG16 is a deep convolutional neural network (CNN) model that is widely used in image classification, feature extraction and other tasks. Therefore, this experiment uses the VGG16 model. The VGG16 model consists of 16 layers, including 13 convolutional layers and 3 fully connected layers.
[0057] (1) VGG16 architecture
[0058] The VGG16 model contains 5 convolution blocks, each of which contains 2 to 3 convolution layers followed by a max pooling layer. There are 3 fully connected layers at the top of the model, and the last layer is a Softmax layer for classification. The specific VGG16 architecture is as follows:
[0059] (a) Convolutional block 1: consists of two convolutional layers (each using 64 3x3 convolutional kernels) followed by a max pooling layer.
[0060] (b) Convolutional block 2: Contains two convolutional layers (each using 128 3x3 convolutional kernels) followed by a max pooling layer.
[0061] (c) Convolutional block 3: Contains three convolutional layers (each layer uses 256 3x3 convolution kernels) followed by a maximum pooling layer.
[0062] (d) Convolutional block 4: Contains three convolutional layers (each using 512 3x3 convolutional kernels) followed by a max pooling layer.
[0063] (e) Convolutional block 5: Contains three convolutional layers (each using 512 3x3 convolutional kernels) followed by a max pooling layer.
[0064] (f) Fully connected layer: It contains two fully connected layers (4096 neurons in each layer) and one fully connected layer (containing 1000 neurons and ultimately outputting 1000 categories).
[0065] (2) Image preprocessing
[0066] Before inputting the image into the VGG16 model, the image needs to be preprocessed. First, the image size is adjusted to 224*224 pixels to adapt to the input requirements of VGG16. Secondly, the image pixel values are normalized and scaled to between 0 and 1 to improve the stability and effect of model training.
[0067] (3) Convolutional layer and pooling layer
[0068] Convolutional layer and pooling layer are the key components of the VGG16 model. The calculation formula is as follows:
[0069] Convolutional layer formula: For the input image I and the convolution kernel K, the output O of the convolution operation is expressed as:
[0070] O(i,j)=(I*K)(i,j)=∑ m ∑ n I(i+m,j+n)K(m,n) (3-4)
[0071] Among them, i and j are the position indexes of the output image, and m and n are the position indexes of the convolution kernel.
[0072] Max pooling layer formula: The max pooling layer calculates the output using the following formula:
[0073]
[0074] Among them, p is the size of the pooling window and s is the pooling stride.
[0075] (4) Feature extraction
[0076] In the VGG16 model, the previous convolutional layers and pooling layers are mainly used to extract image features. The feature extraction process is to pass the image through these layers to obtain a high-dimensional feature representation. Assuming that the size of the input image is 224*224*3 (height*width*channel), after multiple layers of convolution and pooling, the size of the feature map is gradually reduced, and finally a 7*7*512 feature map is obtained after convolution block 5.
[0077] (5) Fully connected layer
[0078] The fully connected layer expands the feature map extracted by the convolutional layer into a vector and performs a linear transformation. Assuming that the input feature map size is 7*7*512, the length of the expanded vector is 7*7*512=25088. The output of the fully connected layer is calculated by the following formula:
[0079] y=Wx+b (3-6)
[0080] Where x is the input vector, W is the weight matrix, and b is the bias vector.
[0081] (6) Vectorization
[0082] In order to achieve the vector representation of the image, the output of the penultimate fully connected layer is intercepted as the vector representation of the image. This layer contains 4096 neurons, so the generated vector representation is 4096 dimensions. This high-dimensional vector can effectively capture the deep features of the image and provide rich information for subsequent image processing and analysis tasks.
[0083] 2 Multimodal Vector Alignment
[0084] This step uses a contrastive learning method to align text and images with the same semantics, so as to shorten the distance between images and texts with the same semantics in the vector space. Taking "apple" as an example, a contrastive learning scheme is used to shorten the vector distance between apple images and apple text. After vector alignment, whether the user's question is text or image, it can accurately match the existing question-answering corpus based on the semantic information of the question to improve the answer accuracy of the large language model.
[0085] 2.1 Symbolic Representation
[0086] With a training data set It contains N samples There are m modes and N corresponding labels These labels come from K categories.
[0087] 2.2 Unimodal Supervised Multimodal Contrastive Learning
[0088] First, unimodal representations are extracted from the raw data of each modality via a pre-trained encoder. A unimodal checking step is introduced to generate weak supervision information to check the validity of each unimodal representation. Then, we show how to design a multimodal contrastive learning method with unimodal supervision between multiple modalities to learn multimodal representations.
[0089] (1) Modal encoder
[0090] Given multimodal training data First, the original unimodal data of modality m is processed by respective encoders to obtain hidden representations. The hidden representation learned by modality m is denoted by r m , in the enterprise documents, VGG16 is used as the feature encoder for images, and Sentence-BERT is used as the feature encoder for the text descriptions in these data.
[0091] (2) Single-mode inspection
[0092] Different from the general aggregate multimodal learning method that only uses single-modal representation for fusion, the method of the present invention also uses single-modal representation as input for single-modal prediction tasks. The classification module can be regarded as a probabilistic model: g φ :R→P, which maps the hidden representation to the predictive distribution P(y|r). For the unimodal prediction task, the predictive distribution is based only on the output of the unimodal classifier. The learning goal of the unimodal prediction task is to minimize the loss of each unimodal prediction:
[0093]
[0094] where y k is the k-th element category label, is the softmax output of the unimodal classifier on modality m.
[0095] The above unimodal prediction results are used to generate supervision information to determine the effectiveness of each modality. Unimodal representations with correct predictions are considered to be effective representations that provide information to the target label. Conversely, unimodal representations with incorrect predictions are considered to be invalid representations.
[0096] (3) Multimodal Contrastive Learning
[0097] The multimodal contrastive learning method proposed in this paper aims to align unimodal representations through weak supervision information. Therefore, a contrastive loss function is designed to effectively align unimodal and multimodal representations. The contrastive loss function is used to bring sample pairs of the same category closer and pull sample pairs of different categories farther apart. Specifically, the following loss function is defined:
[0098]
[0099] in, It represents the similarity between sample i and sample j in mode a and mode b, and τ is the temperature parameter.
[0100] In order to enhance the effect of contrastive learning, the results of the unimodal prediction task are used as weak supervision information. Specifically, the sample pairs with correct unimodal prediction are considered as positive sample pairs, and the sample pairs with incorrect unimodal prediction are considered as negative sample pairs. This can help the model better align the representations of different modalities.
[0101] 2.3 Integration and overall learning objectives
[0102] After completing the multimodal contrastive learning with unimodal supervision, the learned multimodal representation is used for the multimodal classification task. The specific steps are as follows:
[0103] (1) Multimodal representation: The multimodal representation r after contrastive learning c is used as the final multimodal representation.
[0104] (2) Multimodal classifier: Use the multimodal classifier g φc Classify the multimodal representations. The learning goal of the multimodal classification task is to minimize the multimodal classification loss, which is defined as:
[0105]
[0106] Among them, y k is the k-th element category label, is the softmax output of the multimodal classifier.
[0107] (3) Total loss function: The final total loss function combines the unimodal prediction loss and the multimodal contrast loss:
[0108] L total =λ1L uni +λ2L contrast +λ3L multi (3-10)
[0109] Among them, λ1, λ2 and λ3 are weight parameters used to balance the losses of each part. uni represents the unimodal prediction loss, which ensures the quality of each unimodal representation; L contrast is a multimodal contrast loss used to align representations of different modalities; L multi It is a multimodal classification loss, which is used to optimize the final classification performance.
[0110] Through the above method, we can not only effectively align unimodal representations and improve the quality of multimodal representations, but also achieve better performance in multimodal classification tasks.
[0111] 3 Similarity search
[0112] Similarity search is a very important part in many application scenarios, especially in the fields of text analysis and image processing. By calculating the similarity between vectors, we can effectively measure the similarity between different objects, thereby achieving tasks such as classification, clustering, and retrieval. In this experiment, we use vectorized form to represent text and images, and use the cosine similarity algorithm to compare these vectors.
[0113] (1) Cosine similarity formula
[0114] Cosine similarity is a metric used to measure the similarity between two vectors. This algorithm was chosen in this experiment because it works well in high-dimensional space and is insensitive to the size of the vector, only focusing on the direction of the vector. Given two vectors A and B, the cosine similarity calculation formula is as follows:
[0115]
[0116] Among them, A·B represents the dot product of vectors A and B, ‖A‖ is the norm of vector A (that is, the length of the vector), and ‖B‖ is the norm of vector B.
[0117] The specific dot product calculation formula is:
[0118]
[0119] The calculation formula for the vector norm is:
[0120]
[0121] (2) Calculation steps
[0122] First, the dot product of A and B is calculated, then the norm of A and B is calculated, and finally the cosine similarity is calculated using the dot product and norm of A and B. The final result is expressed as:
[0123]
[0124] (3) Return vector
[0125] After calculating the cosine similarity, the most similar vectors can be returned based on the similarity value. Usually, the top k most similar vectors (Top-K) are returned. This is achieved by sorting all the calculated similarity values and selecting the k vectors with the highest similarity. This method is not only simple but also efficient. It can quickly find multiple vectors that are most similar to a given vector in a large-scale data set, providing important support for subsequent analysis and application.
[0126] 4. Design prompts
[0127] When using a large model to organize question-answer pairs and generate answers, the design of prompts is often crucial. Prompts should be clear, specific, and provide enough information so that the model can accurately understand and generate answers. A prompt designed in this experiment is shown in Table 1 below:
[0128] Table 1 Prompt word template
[0129]
[0130] In the prompt template, question is the question, answer is the question-answer pair, and route is the root directory of the image. This prompt allows the large model to give an answer with a markdown-formatted image path, and then convert the markdown-formatted image path into HTML through automated processing and conversion, so that the answer generated by the large language model can be directly embedded in the web page and display the images contained therein.
[0131] Example 2 Experiment and Results
[0132] 1 Experimental Setup
[0133] This experiment was systematically set up and prepared to ensure that the experiment could proceed smoothly and obtain effective results.
[0134] 1.1 Dataset
[0135] This experiment was evaluated using three common datasets: (reference)
[0136] Mscoco_train2014 (refer to Lin TY, Maire M, Belongie S, et al. Microsoft coco: Common objects in context [C] / / Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V13. Springer International Publishing, 2014: 740-755.), this dataset is a picture set, which is widely used in computer vision tasks such as object detection, segmentation and image description.
[0137] OpenEnded_mscoco_train2014_questions (refer to Veit A, Matera T, Neumann L, et al. Coco-text: Dataset and benchmark for text detection and recognition in natural images [J]. arXiv preprint arXiv: 1601.07140, 2016.), this dataset contains open-ended questions with multiple answers corresponding to the images in the Mscoco_train2014 training set.
[0138] Mscoco_train2014_annotations (refer to Sun X, Hu P, Saenko K. Dualcoop: Fastadaptation to multi-label recognition with limited annotations [J]. Advances in Neural Information Processing Systems, 2022, 35: 30569-30582.). This dataset is the annotation dataset of the Mscoco_train2014 training set, which contains detailed image annotation information. These data include question text (question_id), correct answer (multiple_choice_answer) and corresponding image (image_id). As shown in Table 2:
[0139] Table 2 Statistics of the dataset used
[0140]
[0141] 1.2 Environment Construction
[0142] In terms of experimental environment, the server system is Ubuntu 22.04 LTS, equipped with Intel Core i9-13900K processor, graphics card is NVIDIA RTX 4090 24GB, memory is 128GB, and hard drive is 2048GB.
[0143] 1.3 Experimental Configuration
[0144] The present invention adopts a variety of advanced models and tools to ensure the accuracy and effectiveness of the experiment.
[0145] First, for text feature extraction, the pre-trained Sentence-BERT model is used. By inputting the text into the Sentence-BERT model, a high-dimensional text feature vector can be extracted.
[0146] Then, the pre-trained VGG16 model is used to process the image data. By removing the last fully connected layer, the feature vector of the image can be extracted. Next, in order to perform similarity retrieval, the FAISS vector library is selected, which supports multiple index types and search algorithms and can quickly find the most similar vectors in high-dimensional space.
[0147] Finally, in terms of text generation, the locally deployed GLM-9b-Chat model was used (see Du Z, Qian Y, Liu X, et al. Glm: General language model pretraining with autoregressive blankinfilling[J]. arXiv preprint arXiv:2103.10360, 2021.). In this experiment, the results obtained by similarity retrieval were input into the GLM-9b-Chat model to generate the corresponding text answers.
[0148] By combining Sentence-BERT, VGG16, FAISS library and GLM-9b-Chat model, the present invention constructs an efficient and accurate multimodal question-answering system, which provides important guarantee for the smooth progress of the experiment and the reliability of the results.
[0149] 2 Single-modal question-answering experiment
[0150] The goal of the unimodal question-answering experiment is to verify the effectiveness of the Sentence-BERT model in text feature extraction and use the FAISS library for text similarity retrieval.
[0151] First, the text data was segmented, and then Sentence-BERT was used to extract word vectors and average them to obtain text features.
[0152] Next, we created the FAISS vector library to store the extracted text features as vectors for subsequent retrieval. In the similarity retrieval stage, we used the FAISS library for vector retrieval, and returned the three most similar texts for each retrieval.
[0153] Finally, GLM-9b-Chat is used to generate answers, and the correctness of the answers is judged based on the cosine similarity threshold, which is set to 80%. When the similarity of the generated answers exceeds this threshold, the answer is considered correct.
[0154] In order to comprehensively evaluate the performance of the model, a variety of evaluation indicators are used, including accuracy, precision, recall and F1 score. Accuracy measures the proportion of correct answers given by the model; precision measures the accuracy of the correct answers given by the model; recall measures the ability of the model to find all correct answers; and the F1 score is the harmonic average of precision and recall, providing a comprehensive evaluation indicator.
[0155] Through the analysis of experimental results, all evaluation indicators of the model are 0 without using RAG technology (non-RAG). This shows that the model cannot correctly classify or generate effective answers without auxiliary information retrieval. This result can be attributed to the particularity of the dataset, which causes the model to be unable to effectively learn and reason under pure text input. When RAG technology is introduced and only text retrieval (RAG text only) is used, the accuracy of the model is improved to 0.72, the precision reaches 1.00, the recall rate is 0.72, and the F1 score is 0.84. Although the accuracy reaches the maximum value, which means that all answers generated by the model are correct, the recall value indicates that some correct answers have not been retrieved. This may be due to a certain loss in the query of the data vector library, resulting in some data failing to match effectively.
[0156] 3 Multimodal Question Answering Experiment
[0157] The goal of the multimodal question-answering experiment is to combine text data and image data for multimodal feature extraction and similarity retrieval. First, the text data is processed in the same way, and then the image data is preprocessed, the image size is adjusted to 224x224, and the image features are extracted using VGG16. Next, a FAISS vector library is created to store the extracted text and image features as vectors respectively. In the similarity retrieval and model verification stages, the methods are the same as those of the unimodal experiment, and the evaluation indicators also include accuracy, precision, recall, and F1 score. Compared with unimodal question-answering experiments, multimodal question-answering experiments require the simultaneous processing of text and image data, which increases the complexity of the experiment, but also provides a richer source of information, which helps to improve the overall performance of the question-answering system.
[0158] The evaluation indicators without using the RAG method, using the RAG single text method, and using the multimodal method are shown in Table 3:
[0159] Table 3 Experimental results
[0160]
[0161] The experimental results show that the non-RAG model is 0 on all evaluation indicators, which indicates that the model failed to correctly predict any samples and has poor performance. The RAG model that only uses text performs well in accuracy, precision, recall and F1 score, reaching 0.72, 1.00, 0.72 and 0.84 respectively, which shows that the model has good performance in text processing, but there is still room for improvement. When further combining text and image information for retrieval, the performance of the model is significantly improved, with an accuracy of up to 0.87, precision maintained at 1.00, recall reaching 0.87, and F1 score also rising to 0.93. The evaluation indicators have all been significantly improved, with accuracy, recall and F1 scores increased by 15%, 15% and 9% respectively. By introducing image information, the model is able to use more context for retrieval, thereby reducing information loss and improving the overall performance of the model. This result shows that the combination of multimodal information can effectively enhance the understanding and generation capabilities of the model.
[0162] The above contents are merely examples and explanations of the concept of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.
Claims
1. A multimodal question-answering system for enterprise document query combined with retrieval enhancement generation, suitable for text and image queries, characterized in that: It is constructed by combining Retrieval Augmented Generation (RAG) with multimodal technology. It uses algorithms to vectorize existing texts and images and store them in a vector library. It then uses comparative learning to shorten the distance between texts and images with the same semantics in the vector space. It then vectorizes the text and images in the user's question and matches them in the vector library. Finally, it concatenates the matching results with the question, puts them into a prompt, and passes them into the big model to get the answer, thereby realizing multimodal question and answer in enterprise documents.
2. The enterprise document query multimodal question answering system combined with retrieval enhancement generation as claimed in claim 1, characterized in that: Implement the following steps: First, the Sentence-BERT model is used to vectorize the text to capture the semantic relationship between words and generate high-quality text vectors. At the same time, the VGG16 model is selected to vectorize the image to extract image features. Next, the vectors of these texts and images are stored using the FAISS vector library, which has efficient vector retrieval capabilities and can quickly find and match high-dimensional vectors. Then, by comparing texts and pictures with the same semantics, the distance between them in the vector space is shortened to achieve better cross-modal understanding and representation learning; the user's questions will be matched in the vector library through the cosine similarity algorithm to find the most relevant question-answer pairs; During the matching process, the Top-K most similar question-answer pairs are returned to ensure that diverse and relevant answers are provided; Finally, the user's question and the matched question-answer pair are embedded in the designed prompt word template and passed to the big model for analysis and answering. In this way, the big model can comprehensively consider the user's question and related context to generate accurate and detailed answers.
Citation Information
Patent Citations
Multi-modal implication enhanced image text retrieval method and system
CN115712740A
Agricultural intelligent question and answer method and system supporting multi-mode and multi-round dialogues
CN117370506A
Large model image-text dialogue method and device, equipment and storage medium
CN117633173A
Cited By
Hybrid retrieval method, device and equipment based on industrial large model and medium
CN119293269A
Query method, device and system, storage medium and computer program product
CN119415781A
Multi-modal auxiliary precision manufacturing database interaction method based on large model
CN120104817A
Intelligent question answering method and system based on question index retrieval enhancement generation technology
CN120124640A
Document question and answer method and device, storage medium and program product
CN121597789A
