Question and answer method and system based on image-text multi-modal enhanced retrieval and electronic equipment
By adopting multimodal enhancement retrieval method in the question and answer system, using vectorization technology and large language model, the problem of poor accuracy of the existing question and answer system is solved, and a more accurate and effective question and answer process is achieved.
Patent Information
- Application Number
- CN202411770973.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-03-25
AI Technical Summary
The existing question and answer system is poor in accuracy, making it difficult for users to obtain satisfactory answers, mainly because users cannot accurately express their problem needs.
Using a question-and-answer method based on multimodal enhancement retrieval of graphics and text, the text vector model and graphic vector model are trained, and the text and picture content are vectorized and stored in the vector database. After obtaining the user's question text, it searches, generates important text and inputs a large language model to generate replies.
It improves the accuracy of the question-and-answer system, and can generate corresponding prompt words by retrieving image information, thereby providing more accurate answers.
Smart Images
Figure CN119691122A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a question-answering method and system based on multimodal enhanced retrieval of images and texts, and an electronic device. Background Art
[0002] Question answering system (QA) is an advanced form of information retrieval system. It can answer questions raised by users in natural language accurately and concisely. It is an important path to artificial intelligence. The current question answering system usually uses a large language model as the basic model. The large language model (LLM) is a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. The question answering system uses a large language model to handle a variety of natural language tasks, such as text classification, question answering, dialogue, etc., and has a wide range of applications, including: smart home, medical care, education, Internet of Vehicles, etc.
[0003] Traditional question-answering systems based on large language models usually filter answers based on information retrieval. The main process is: the system first understands the user's question through natural language processing technology. This includes word segmentation, grammatical analysis, entity recognition, etc., in order to accurately grasp the user's intention. Then the system retrieves information related to the user's question from the knowledge base. The knowledge base can be a large text database containing knowledge in various fields. The system finds relevant answers by matching keywords, context, etc. Finally, the system generates answers based on the retrieved information. This can be done by generating answers from scratch or extracting answers from the knowledge base and slightly modifying them. However, since users often cannot accurately express their own question requirements, the accuracy of question and answer is poor, and it is difficult for users to get satisfactory answers. Summary of the invention
[0004] The present invention aims to solve the problem of poor accuracy in existing question-answering systems, and proposes a question-answering method and system, and electronic equipment based on image-text multimodal enhanced retrieval.
[0005] The technical solution adopted by the present invention to solve the above technical problems is:
[0006] In a first aspect, the present invention provides a question-answering method based on multimodal enhanced retrieval of images and texts, the method comprising:
[0007] Training a text vector model and a picture-text vector model, using the text vector model to vectorize the text content to obtain a vector corresponding to the text content, using the picture-text vector model to vectorize the picture content to obtain a vector corresponding to the picture content, and storing the vector corresponding to the text content and the vector corresponding to the picture content in a vector database;
[0008] Obtain the question text requested by the user, vectorize the question text using a text vector model, obtain a vector corresponding to the question text, search the vector database according to the vector corresponding to the question text, and obtain the corresponding first K text information and picture information respectively;
[0009] The text information is reorganized, summarized and rearranged to obtain the first N important texts, corresponding important texts are obtained according to the picture information, the important texts are encapsulated as prompt words and then input into the large language model to obtain a streaming reply result.
[0010] Furthermore, the text vector model is trained, including:
[0011] A basic text vector model is obtained, and the positive and negative sample data corresponding to the text content are converted into JSON format data, the basic text vector model is fine-tuned using the JSON format data to obtain a fine-tuned model, and the fine-tuned model is weightedly merged with the basic text vector model to obtain a text vector model.
[0012] Furthermore, the image-text vector model is trained, including:
[0013] A basic image vector model is obtained, and based on the text vector model and the basic image vector model, vectorized representations of text and image are mapped to the same joint channel space to generate text-image pairs, and a multimodal model is trained based on the text-image pairs to obtain a text-image vector model based on the text vector model.
[0014] Furthermore, the text content includes user information, text-graphic knowledge base and text content corresponding to the knowledge graph, and the image content includes image content corresponding to the text-graphic knowledge base.
[0015] Furthermore, the text content corresponding to the knowledge graph includes entity triples, and vectorizing the entity triples includes:
[0016] The entity information of the entity triplet is taken as the first dimension of the vector array, the attribute information of the entity triplet is taken as the second dimension of the vector array, and the relationship information of the entity triplet is taken as the third dimension of the vector array. The information corresponding to each vector array dimension is respectively vectorized and weighted to obtain an entity vector with the same dimension as the vector array.
[0017] Furthermore, the method further comprises:
[0018] After obtaining the question text of the user's request, sensitive word detection is performed on the question text. If sensitive words are present, the user's request is blocked.
[0019] Furthermore, the method further comprises:
[0020] After obtaining the streaming reply result, the streaming reply result is detected for sensitive words, and the detected sensitive words are blocked.
[0021] In a second aspect, the present invention provides a question-answering system based on multimodal enhanced retrieval of images and texts, the system comprising:
[0022] A vectorization unit, used to train a text vector model and a picture-text vector model, vectorize text content using the text vector model to obtain a vector corresponding to the text content, vectorize picture content using the picture-text vector model to obtain a vector corresponding to the picture content, and store the vector corresponding to the text content and the vector corresponding to the picture content in a vector database;
[0023] A retrieval unit is used to obtain the question text requested by the user, vectorize the question text using a text vector model, obtain a vector corresponding to the question text, and search the vector database according to the vector corresponding to the question text to obtain the first K corresponding text information and picture information;
[0024] The answer generation unit is used to reorganize, extract and rearrange the text information to obtain the first N important texts, obtain the corresponding important texts according to the image information, input the important texts into the large language model, and obtain a streaming reply result.
[0025] In a third aspect, the present invention provides an electronic device, the electronic device comprising a processor, a memory and a communication bus;
[0026] The communication bus is used to realize the connection and communication between the processor and the memory;
[0027] The processor is used to execute one or more programs in the memory to implement the steps of the question-answering method based on image-text multimodal enhanced retrieval as described in the first aspect.
[0028] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed, implements the steps of the question-answering method based on multimodal enhanced retrieval of images and texts as described in the first aspect.
[0029] The beneficial effects of the present invention are as follows: the question-answering method and system based on multimodal enhanced retrieval of text and images provided by the present invention respectively vectorizes the text content and image content and stores them in a vector database. After obtaining the question text requested by the user, the corresponding text information and image information are respectively retrieved from the vector database, and an answer is generated based on the retrieved text information and image information. When retrieving information, the present invention not only retrieves the corresponding text information based on the question text, but also retrieves the corresponding image information based on the question text, and generates important text based on the retrieved text information and image information. After the important text is encapsulated with prompt words, it is input into the language model to generate the corresponding reply answer. The present invention realizes enhanced retrieval of multimodal knowledge of text and images, thereby improving the accuracy of question and answer. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 A flowchart of a question-answering method based on multimodal enhanced retrieval of images and texts provided in an embodiment;
[0031] Figure 2 A schematic diagram of the structure of a question-answering system based on multimodal enhanced retrieval of images and texts provided in an embodiment;
[0032] Figure 3 A schematic structural diagram of an electronic device provided in an embodiment. DETAILED DESCRIPTION
[0033] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution in this embodiment will be clearly and completely described below in conjunction with the drawings in this embodiment.
[0034] In some of the processes described in the specification of the present invention and the above-mentioned drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The sequence numbers of the operations are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.
[0035] At present, the implementation process of the question-answering system based on the large language model is usually as follows: first, the user's question is understood through natural language processing technology, then the information related to the user's question is retrieved from the knowledge base, and finally the answer is generated based on the retrieved information. After research, the inventor found that this method has poor accuracy in question-answering because the user cannot accurately express his or her own question needs.
[0036] Based on this, the technical solution of the present invention is proposed. In the present invention, by training a text vector model and a picture-text vector model, the text content is vectorized by using the text vector model to obtain a vector corresponding to the text content, the picture content is vectorized by using the picture-text vector model to obtain a vector corresponding to the picture content, and the vector corresponding to the text content and the vector corresponding to the picture content are stored in a vector database; the question text requested by the user is obtained, the question text is vectorized by using the text vector model to obtain a vector corresponding to the question text, the vector database is searched according to the vector corresponding to the question text, and the corresponding first K text information and picture information are respectively obtained; the text information is reorganized, the summary is extracted and rearranged to obtain the first N important texts, the corresponding important texts are obtained according to the picture information, the important texts are encapsulated as prompt words and then input into the large language model to obtain a streaming reply result.
[0037] Specifically, the present invention vectorizes the text content and the image content respectively and stores them in a vector database. After obtaining the question text requested by the user, the vector database is searched according to the vector of the question text. Not only the corresponding text information is retrieved according to the question text, but also the corresponding image information is retrieved according to the question text. Important text is generated based on the retrieved text information and image information, and after the important text is encapsulated with prompt words, it is input into the language model to generate the corresponding reply answer. The present invention realizes multi-modal image-text enhanced retrieval. Even if the user does not accurately express the question requirements, the corresponding prompt words can be generated by retrieving the corresponding image information, thereby improving the accuracy of the question and answer.
[0038] The technical solution in this embodiment will be clearly and completely described below in conjunction with the drawings in this embodiment. Obviously, the described embodiment is only a part of the embodiments of the present invention, rather than all the embodiments.
[0039] Figure 1 A flowchart of a question-answering method based on multimodal enhanced retrieval of images and texts is shown, see Figure 1 , the method comprises the following steps:
[0040] S1. Train a text vector model and a picture-text vector model, use the text vector model to vectorize the text content to obtain a vector corresponding to the text content, use the picture-text vector model to vectorize the image content to obtain a vector corresponding to the image content, and store the vector corresponding to the text content and the vector corresponding to the image content in a vector database.
[0041] In this embodiment, training a text vector model specifically includes: obtaining a basic text vector model, and converting positive and negative sample data corresponding to the text content into JSON format data, using the JSON format data to fine-tune the basic text vector model to obtain a fine-tuning model, and weightedly merging the fine-tuning model with the basic text vector model to obtain a text vector model.
[0042] In practical applications, the basic text vector model can be a BGE model, a BCE model, etc. The JSON format data of the positive and negative sample data corresponding to the text content is: {"query":str,"pos":List[str],"neg":List[str]}. By weighted merging of the fine-tuning model with the basic text vector model, the model performance can be improved, and the performance of the model in specific fields and general tasks can be balanced.
[0043] In this embodiment, training a graphic-text vector model specifically includes: obtaining a basic image vector model, mapping the vectorized representations of text and image to the same joint channel space based on the text vector model and the basic image vector model, generating text-image pairs, training a multimodal model based on the text-image pairs, and obtaining a graphic-text vector model based on the text vector model.
[0044] In practical applications, based on the obtained text vector model and the basic image vector model, the vectorized representations of text and image are mapped to the same joint channel space. Then the cosine similarity between the image vector and the text vector is calculated, and the objective function of contrastive learning is set to make the similarity of positive sample pairs higher and the similarity of negative sample pairs lower. Finally, the contrastive loss function is used to calculate the model weights to obtain the image-text vector model based on the text vector model.
[0045] In this embodiment, the text content includes user information, text content corresponding to the graphic and text knowledge base and the knowledge graph, and the image content includes image content corresponding to the graphic and text knowledge base.
[0046] Among them, the graphic knowledge base is a database form that combines text and image information, and is mainly used to manage and store structured data. The graphic knowledge base not only contains traditional text information, but also manages image files. The knowledge graph is a graphical data model used to represent and organize structured knowledge. The text content corresponding to the knowledge graph includes entity triples.
[0047] In practical applications, first, the entity triples of the knowledge graph are vectorized, specifically including: taking the entity information of the entity triple as the first dimension of the vector array, taking the attribute information of the entity triple as the second dimension of the vector array, taking the relationship information of the entity triple as the third dimension of the vector array, and respectively vectorizing the information corresponding to each vector array dimension and weighting it to obtain an entity vector with the same dimension as the vector array.
[0048] Then, the text content corresponding to the user information and the graphic knowledge base is vectorized. The text content includes structured text content and text content corresponding to the user information. For the structured text content, the text vector model is used to convert the text content into natural language text, and the natural language text in the text content is recognized as an entity to obtain the vector corresponding to the structured text content, including the knowledge text vector and the entity vector, which are finally stored in the vector database. For the text content corresponding to the user information, the text is first sliced, and then semantic entity recognition is performed on each paragraph of text to obtain the vector of the text content corresponding to the user information, including the vector of each paragraph of text and the entity vector, which are finally merged and stored in the vector database.
[0049] Finally, the image content is vectorized using the image-text vector model. First, the image vector is mapped to the same text vector space based on the multimodal algorithm, and then the entity information contained in the image is identified using vector retrieval to obtain the vector corresponding to the image content, including the image vector and the entity vector, which are finally merged and stored in the vector database.
[0050] S2. Obtain the question text requested by the user, vectorize the question text using a text vector model, obtain the vector corresponding to the question text, search the vector database according to the vector corresponding to the question text, and obtain the corresponding first K text information and picture information respectively.
[0051] In this embodiment, after obtaining the question text requested by the user, the question text is detected for sensitive words. If there are sensitive words, the user request is blocked. If there are no sensitive words, the question text is vectorized, and the vector database is searched according to the vector corresponding to the question text. Specifically, it can be: according to the vector corresponding to the question text and based on the int8_hnsw search algorithm, the TOP-K text information is first retrieved from the vector database, and then the TOP-K image information is retrieved. Through sensitive word detection, the security of the question-answering system can be improved.
[0052] S4. Reorganize, extract and rearrange the text information to obtain the first N important texts, obtain the corresponding important texts according to the picture information, encapsulate the important texts into prompt words and input them into the large language model to obtain a streaming reply result.
[0053] For the TOP-K text information, these text information can be reorganized, summarized, integrated and classified, and then rearranged to obtain the TOP-N important texts, and then the important texts are input into the language model to obtain streaming replies. For the TOP-K image information, the important text corresponding to the image is obtained based on the image information, and then the important text is input into the language model to obtain streaming replies. After obtaining the streaming reply results, it also includes sensitive word detection for the streaming reply results and blocking the detected sensitive words.
[0054] By reorganizing, summarizing and rearranging text information, the amount of data input into the large language model can be reduced, thereby increasing the speed at which the large language model generates responses and avoiding interference from useless information on the results, thereby improving the efficiency and accuracy of question-answering.
[0055] In summary, the question-answering method based on multimodal enhanced retrieval of images and texts provided in this embodiment vectorizes the text content and the image content respectively and stores them in a vector database. After obtaining the question text requested by the user, the vector database is searched according to the vector of the question text. Not only the corresponding text information is retrieved according to the question text, but also the corresponding image information is retrieved according to the question text. Important texts are generated based on the retrieved text information and image information. After the important texts are encapsulated with prompt words, they are input into the language model to generate corresponding reply answers. This embodiment realizes multimodal enhanced retrieval of images and texts. Even if the user does not accurately express the question requirements, the corresponding prompt words can be generated by retrieving the corresponding image information, thereby improving the accuracy of questions and answers. In addition, this embodiment can reduce the amount of data input into the language model by reorganizing, extracting and rearranging the text information, thereby improving the reply generation speed of the language model and avoiding the interference of useless information on the results, thereby improving the efficiency and accuracy of questions and answers.
[0056] Figure 2 A structural diagram of a question-answering system based on multimodal enhanced retrieval of images and texts is shown in FIG. Figure 2 , the system comprising:
[0057] A vectorization unit, used to train a text vector model and a picture-text vector model, vectorize text content using the text vector model to obtain a vector corresponding to the text content, vectorize picture content using the picture-text vector model to obtain a vector corresponding to the picture content, and store the vector corresponding to the text content and the vector corresponding to the picture content in a vector database;
[0058] A retrieval unit is used to obtain the question text requested by the user, vectorize the question text using a text vector model, obtain a vector corresponding to the question text, and search the vector database according to the vector corresponding to the question text to obtain the first K corresponding text information and picture information;
[0059] The answer generation unit is used to reorganize, extract and rearrange the text information to obtain the first N important texts, obtain the corresponding important texts according to the image information, input the important texts into the large language model, and obtain a streaming reply result.
[0060] It can be understood that since the question and answer system based on multi-modal enhanced retrieval of images and texts described in this embodiment is a system for implementing the question and answer method based on multi-modal enhanced retrieval of images and texts described in the embodiment, for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method, which will not be repeated here.
[0061] Figure 3 A schematic diagram of the structure of an electronic device is shown. Figure 3 , the electronic device includes a processor, a memory and a communication bus;
[0062] The communication bus is used to realize the connection and communication between the processor and the memory;
[0063] The processor is used to execute one or more programs in the memory to implement the steps of the question-answering method based on multi-modal enhanced retrieval of images and texts described in this embodiment.
[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
Claims
1. A question answering method based on multimodal enhanced retrieval of images and texts, characterized in that: The method comprises: Training a text vector model and a picture-text vector model, using the text vector model to vectorize the text content to obtain a vector corresponding to the text content, using the picture-text vector model to vectorize the picture content to obtain a vector corresponding to the picture content, and storing the vector corresponding to the text content and the vector corresponding to the picture content in a vector database; Obtain the question text requested by the user, vectorize the question text using a text vector model, obtain a vector corresponding to the question text, search the vector database according to the vector corresponding to the question text, and obtain the corresponding first K text information and picture information respectively; The text information is reorganized, summarized and rearranged to obtain the first N important texts, corresponding important texts are obtained according to the picture information, the important texts are encapsulated as prompt words and then input into the large language model to obtain a streaming reply result.
2. The question-answering method based on multimodal enhanced retrieval of images and texts according to claim 1, characterized in that: Training text vector models, including: A basic text vector model is obtained, and the positive and negative sample data corresponding to the text content are converted into JSON format data, the basic text vector model is fine-tuned using the JSON format data to obtain a fine-tuned model, and the fine-tuned model is weightedly merged with the basic text vector model to obtain a text vector model.
3. The question-answering method based on multimodal enhanced retrieval of images and texts according to claim 2 is characterized in that: Training image-text vector models, including: A basic image vector model is obtained, and based on the text vector model and the basic image vector model, vectorized representations of text and image are mapped to the same joint channel space to generate text-image pairs, and a multimodal model is trained based on the text-image pairs to obtain a text-image vector model based on the text vector model.
4. The question-answering method based on multimodal enhanced retrieval of images and texts according to claim 1, characterized in that: The text content includes user information, text content corresponding to the graphic knowledge base and the knowledge graph, and the image content includes image content corresponding to the graphic knowledge base.
5. The question-answering method based on multimodal enhanced retrieval of images and texts according to claim 4 is characterized in that: The text content corresponding to the knowledge graph includes entity triples, and vectorizing the entity triples includes: The entity information of the entity triplet is taken as the first dimension of the vector array, the attribute information of the entity triplet is taken as the second dimension of the vector array, and the relationship information of the entity triplet is taken as the third dimension of the vector array. The information corresponding to each vector array dimension is respectively vectorized and weighted to obtain an entity vector with the same dimension as the vector array.
6. The question-answering method based on multimodal enhanced retrieval of images and texts according to claim 1, characterized in that: The method further comprises: After obtaining the question text of the user's request, sensitive word detection is performed on the question text. If sensitive words are present, the user's request is blocked.
7. The question-answering method based on multimodal enhanced retrieval of images and texts according to claim 1, characterized in that: The method further comprises: After obtaining the streaming reply result, the streaming reply result is detected for sensitive words, and the detected sensitive words are blocked.
8. A question-answering system based on multimodal enhanced retrieval of images and texts, characterized in that: The system comprises: A vectorization unit, used to train a text vector model and a picture-text vector model, vectorize text content using the text vector model to obtain a vector corresponding to the text content, vectorize picture content using the picture-text vector model to obtain a vector corresponding to the picture content, and store the vector corresponding to the text content and the vector corresponding to the picture content in a vector database; A retrieval unit is used to obtain the question text requested by the user, vectorize the question text using a text vector model, obtain a vector corresponding to the question text, and search the vector database according to the vector corresponding to the question text to obtain the first K corresponding text information and picture information; The answer generation unit is used to reorganize, extract and rearrange the text information to obtain the first N important texts, obtain the corresponding important texts according to the image information, input the important texts into the large language model, and obtain a streaming reply result.
9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a communication bus; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to execute one or more programs in the memory to implement the steps of the question-answering method based on image-text multimodal enhanced retrieval as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the steps of the question-answering method based on image-text multimodal enhanced retrieval are implemented as described in any one of claims 1 to 7.