Multi-mode intelligent question answering method and device, storage medium and computer equipment

Abstract

According to the multi-modal intelligent question answering method and device, the storage medium and the computer equipment, when the query question is received, the target query text of the query question is extracted to avoid input carrier limitation of the query question, then the target query text is vectorized into the target text vector, and the query question is queried according to the target text vector. The target text block is obtained based on the target query text and the target text vector, so that the retrieval accuracy is improved; and then optimizing the target text block by adopting the target text optimization model and outputting a high-accuracy target answer. If the target answer contains the picture address, a picture request is initiated to the server after a request address corresponding to the picture address is generated, and the server performs security verification on the request address and returns picture content corresponding to the picture address after the request address passes the security verification, so that the data security is improved; and finally, replacing the picture address in the target answer with the picture content, and displaying the final target answer to the user through a multi-modal mode of the text and the picture, so that the intuition and the accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a multimodal intelligent question-answering method, apparatus, storage medium, and computer equipment. Background Art

[0002] Currently, the RAG (Retrieval-Augmented Generation) + LLM (Large Language Model) framework is one of the mainstream technologies for knowledge-based intelligent question answering. Specifically, RAG technology combines information retrieval and text generation. It searches for task-related materials in large-scale document libraries and uses these materials to guide answer generation or text construction, thereby improving the accuracy and reliability of LLM model output.

[0003] However, the input and output of the current RAG+LLM framework are strictly limited to the text range, and the number of tokens is also limited, which makes it impossible for the LLM model to directly parse images. Many important visual information carriers, such as charts, flow charts, application forms and tables, are difficult to be fully and accurately converted into text form, which leads to omissions or misunderstandings in input and output information, affecting the intuitiveness and accuracy of the output answers. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defect that the intelligent question-answering method in the prior art cannot directly parse the image, resulting in omission or misunderstanding of the input and output information, affecting the intuitiveness and accuracy of the output answer.

[0005] The present application provides a multimodal intelligent question answering method, the method comprising:

[0006] Receiving a query question input by a user, extracting a target query text of the query question, and vectorizing the target query text to generate a target text vector;

[0007] Retrieving multiple target text blocks from a knowledge base based on the target query text and the target text vector, and optimizing each target text block using a target text optimization model to obtain a target answer to the query question;

[0008] When it is detected that the target answer includes a picture address, generating a request address based on the picture address and the server address; the request address includes verification information corresponding to the picture address;

[0009] Initiate a picture request to the corresponding server through the request address, so that the server performs a security check on the verification information in the request address, and returns the picture content corresponding to the picture address after the verification passes;

[0010] The image address in the target answer is replaced with the image content, and the target answer is updated.

[0011] Optionally, the extracting a target query text of the query question includes:

[0012] Determine whether there are other visual information carriers besides text content in the query question; the other visual information carriers include pictures and tables:

[0013] If yes, then identify the text information of the other visual information carriers, and perform key text extraction on the text content and text information to obtain the target query text of the query question;

[0014] If not, the key text of the text content is directly extracted as the target query text of the query question.

[0015] Optionally, the knowledge base pre-stores various text blocks of a plurality of search documents and a search vector corresponding to each text block;

[0016] The retrieving a plurality of target text blocks from a knowledge base based on the target query text and the target text vector comprises:

[0017] Performing similarity matching between the target query text and each text block in the knowledge base to obtain a first matching result, and performing similarity matching between the target text vector and each search vector in the knowledge base to obtain a second matching result;

[0018] Determining the similarity of each text block according to the first matching result and the second matching result, and sorting the text blocks according to the similarities to obtain a sorting result;

[0019] According to a preset similarity threshold and a preset number of selections, a plurality of target text blocks are obtained by screening the text blocks in the sorting results.

[0020] Optionally, the process of constructing the knowledge base includes:

[0021] Acquire a search document from multiple data sources, and perform text analysis on the search document to obtain a analysis result; the analysis result includes text content and other visual information carriers;

[0022] Extracting key texts from other visual information carriers, converting the other visual information carriers into picture formats, and storing them after forming picture contents;

[0023] Generate the image address of the other visual information carrier according to the storage ID of the image content and the key text;

[0024] Replacing other visual information carriers in the search document with corresponding image addresses, and performing text segmentation on the replaced search document to obtain multiple text blocks;

[0025] A search vector corresponding to each text block is generated, and a knowledge base is constructed based on each text block and each search vector.

[0026] Optionally, the adopting the target text optimization model to perform text optimization on each target text block to obtain a target answer to the query question includes:

[0027] Determine a large language model prompt word for the query question;

[0028] The large language model prompt words and each target text block are input into the target text optimization model, so that the target text optimization model performs context interpretation on each target text block based on the large language model prompt words, and performs text optimization on each target text block according to the interpretation results to obtain the target answer to the query question.

[0029] Optionally, the process of generating the verification information includes:

[0030] After retrieving the target text block of the query question from the knowledge base, traversing to obtain the image address in each target text block;

[0031] Determine the retrieval time and validity period of the query question, and the storage ID in the image address, and use a signature generation algorithm to encrypt and calculate the retrieval time, the validity period and the storage ID to obtain a first signature;

[0032] Verification information of the image address is generated according to the retrieval time, the validity period and the signature.

[0033] Optionally, the security verification process of the verification information includes:

[0034] Extract the retrieval time, validity period and first signature in the verification information, as well as the storage ID and current time in the image address, and use the signature generation algorithm to encrypt and calculate the retrieval time, validity period and storage ID to obtain a second signature;

[0035] Determine whether the time difference between the retrieval time and the current time is within the validity period and the second signature is consistent with the first signature;

[0036] If yes, then confirm that the verification has passed;

[0037] If not, the verification fails.

[0038] The present application also provides a multi-modal intelligent question-answering device, comprising:

[0039] A retrieval data extraction module is used to receive a query question input by a user, extract a target query text of the query question, and vectorize the target query text to generate a target text vector;

[0040] An answer generation module, used to retrieve multiple target text blocks from a knowledge base based on the target query text and the target text vector, and use a target text optimization model to perform text optimization on each target text block to obtain a target answer to the query question;

[0041] A request address generation module, configured to generate a request address based on the image address and the server address when it is detected that the target answer includes an image address; the request address includes verification information corresponding to the image address;

[0042] The image content acquisition module is used to initiate an image request to the corresponding server through the request address, so that the server performs a security check on the verification information in the request address, and returns the image content corresponding to the image address after the verification passes;

[0043] The answer updating module is used to replace the picture address in the target answer with the picture content and update the target answer.

[0044] The present application also provides a storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multimodal intelligent question-answering method as described in any one of the above embodiments.

[0045] The present application also provides a computer device, comprising: one or more processors, and a memory;

[0046] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the multimodal intelligent question-answering method described in any one of the above embodiments are performed.

[0047] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0048] The multimodal intelligent question-answering method, apparatus, storage medium, and computer device provided by the present application can, upon receiving a query question input by a user, extract the target query text of the query question to avoid the input carrier limitation of the query question, and then vectorize the target query text to generate a target text vector, and retrieve multiple target text blocks from the knowledge base based on the target query text and the target text vector, thereby improving the semantic relevance of the text blocks and improving the retrieval accuracy; then, a target text optimization model can be used to optimize the text of each target text block, and the model can reduce the ambiguity of the text by controlling the output direction of the text, and output a target answer with high compliance and accuracy. In addition, when the target answer contains an image address, the present application can generate a request address based on the image address and the server address. Since the request address includes verification information corresponding to the image address, after initiating an image request to the corresponding server through the request address, the server can perform a security check on the verification information in the request address and return the image content corresponding to the image address after the verification passes, thereby preventing malicious requests and illegal access and improving data security; finally, the present application can replace the image address in the target answer with the image content, so that the final target answer can be displayed to the user through a multimodal mode of text and images, thereby improving the intuitiveness and accuracy of the target answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A flowchart of a multimodal intelligent question-answering method provided in an embodiment of the present application;

[0051] Figure 2 A flowchart of a target text block retrieval process provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a process of building a knowledge base provided in an embodiment of the present application;

[0053] Figure 4 A flow diagram of a multimodal intelligent question-answering logic provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of the structure of a multi-modal intelligent question-answering device provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] Currently, the input and output of the RAG+LLM framework are strictly limited to the text range, and the number of tokens is also limited, which makes it impossible for the LLM model to directly parse images. Many important visual information carriers, such as charts, flow charts, application forms and tables, are difficult to be fully and accurately converted into text form, which leads to omissions or misunderstandings in input and output information, affecting the intuitiveness and accuracy of the output answers.

[0058] In one embodiment, Figure 1 As shown, Figure 1 A flowchart of a multimodal intelligent question-answering method provided in an embodiment of the present application; the present application provides a multimodal intelligent question-answering method, which specifically includes the following:

[0059] S110: receiving a query question input by a user, extracting a target query text of the query question, and vectorizing the target query text to generate a target text vector.

[0060] In this step, when the user is conducting a knowledge base question and answer session, the user can enter a corresponding query question in the interactive interface of the computer device. After receiving the query question entered by the user, the computer device can extract the corresponding target query text from the query question. Then, the computer device can vectorize the target query text to generate a target text vector, and then retrieve multiple target text blocks from the knowledge base based on the target query text and the target text vector, thereby improving the semantic relevance of the text blocks and improving the retrieval accuracy.

[0061] It is understandable that the input format of the query question here can be not only text format, but also image, table and other formats, and can also be a combination of multiple formats, which is not limited here. Therefore, this application can support the input of questions in multiple formats, thereby enhancing the expression ability of query questions and query accuracy, thereby improving the ability to process complex information and expanding the application scenarios of intelligent question-answering methods.

[0062] Specifically, after receiving the query question input by the user, the computer device can first parse the query question, so that the target query text corresponding to the query question can be extracted from the parsing result; the target query text can represent key information such as key sentences or keywords in the query question, and these key information reflects the user's query intention. In addition, the computer device can also vectorize the target query text to generate a target text vector, which can be converted into a high-dimensional vector through natural language processing (Natural Language Processing, NLP) and other technologies to represent the position and characteristics of the target query text in the semantic space, providing rich semantic information for the subsequent retrieval process. Therefore, through the target query text and the target text vector, the present application can significantly improve the relevance of the retrieval results, reduce the errors or irrelevant information that may be generated based on keyword matching in the traditional method, so that users can obtain the required information more quickly and accurately.

[0063] S120: Retrieve multiple target text blocks from the knowledge base based on the target query text and the target text vector, and use the target text optimization model to perform text optimization on each target text block to obtain a target answer to the query question.

[0064] In this step, after obtaining the target query text and the target text vector in step S110, the computer device can retrieve multiple target text blocks from the knowledge base based on the target query text and the target text vector. In addition, since the target text optimization model can reduce the ambiguity of the text by controlling the output direction of the text and output a target answer with high compliance and accuracy, the computer device can also use the target text optimization model to optimize the text of each target text block, and finally obtain the target answer to the query question.

[0065] It is understandable that vector retrieval mainly identifies text blocks related to the target text vector through semantic matching, but when faced with longer texts, the quality of the corresponding vector will decrease, resulting in inaccurate calculation of semantic relevance; while text retrieval can focus on accurate matching based on keywords and phrases to ensure that every detail of the query can be responded to. Therefore, this application can combine the target query text and the target text vector to retrieve text blocks, and use text retrieval to make up for the lack of precision of vector retrieval, thereby improving the relevance and accuracy of the retrieval results.

[0066] Specifically, in order to further improve the accuracy and compliance of the answers, the computer device can also introduce a target text optimization model to refine and optimize each target text block. After the computer device inputs each target text block into the target text optimization model, the model can analyze and adjust the content of the text block through a variety of natural language processing techniques to ensure that the output target answer meets the user's query intent. In detail, the main function of the target text optimization model is to control the output direction, style and tone of the text, and interpret and optimize the text content in the target text block in combination with contextual information, thereby reducing the ambiguity of the output results and providing users with reliable and clear target answers.

[0067] S130: When it is detected that the target answer includes a picture address, a request address is generated based on the picture address and the server address; the request address includes verification information corresponding to the picture address.

[0068] In this step, after obtaining the target answer through step S120, the computer device can perform text detection on the target answer, and when it is detected that the target answer contains a picture address, a request address is generated based on the picture address and the server address, and then the corresponding picture content can be obtained through the request address to achieve multi-formatting of the target answer and improve intuitiveness and accuracy.

[0069] It is understandable that in order to improve the information integrity of the knowledge base, the knowledge base can construct text blocks in the form of text and image addresses, where each image address corresponds to an image content. By directly outputting the image content, it is possible to avoid the errors or omissions that are prone to occur in the prior art when extracting text information of the image content during the pure text output process. Based on this, in this application, when the user conducts intelligent question and answering of the knowledge base, the computer device can output the corresponding text information and / or image content as the target answer according to the query question input by the user, thereby realizing the multi-modal interaction of the knowledge base intelligent question and answer, and reducing the complexity of the interaction through the image content.

[0070] Specifically, after generating the target answer, the computer device can perform text detection on the answer, and when it detects that the target answer contains an image address, it automatically generates a request address based on the image address and the preset server address. The request address is the access entry for the image content resource. The computer device sends a request to the server through it to obtain the image content corresponding to the image address. In this way, the computer device can not only provide answers at the text level, but also embed the corresponding image content directly into the target answer, so that users can obtain more intuitive visual information.

[0071] Furthermore, when generating the request address, the computer device can also synchronously generate verification information corresponding to the image address and embed it into the request address. The verification information usually contains information such as an encryption token, validity period, and signature, in order to ensure the legitimacy of the request and prevent potential malicious requests or illegal access. Through the verification information, the server that receives the request address can verify the legitimacy and validity of the request source, ensuring that only authorized users or devices can access and obtain specific image content.

[0072] S140: Initiate a picture request to the corresponding server through the request address, so that the server performs a security check on the verification information in the request address, and returns the picture content corresponding to the picture address after the verification passes.

[0073] In this step, after the request address is generated through step S130, the computer device can initiate an image request to the corresponding server through the request address, so that the server can perform a security check on the verification information in the request address and return the image content corresponding to the image address after the verification passes, thereby preventing malicious requests and illegal access and improving data security.

[0074] Specifically, when a computer device initiates an image request to a server through a request address, in order to ensure the security of the request, the computer device will send a request address containing verification information to the server. After receiving the request, the server will first perform a strict security check on the verification information in the request address. If the verification information is valid and meets the security standards, the server will return the image content corresponding to the image address. This verification process effectively prevents unauthorized access and reduces the risk of attacks, such as security threats such as malicious users forging requests, cross-site request forgery (CSRF) or man-in-the-middle attacks (MITM), thereby protecting the integrity and security of the data.

[0075] S150: Replace the image address in the target answer with the image content, and update the target answer.

[0076] In this step, after obtaining the image content corresponding to the image address through step S140, the computer device can use the image content to replace the image address in the target answer, so that the target answer with both pictures and text can be obtained. In this way, the computer device can realize the multi-format presentation of the target answer, so that the answer is not limited to text descriptions, but integrates multiple visual information carriers such as pictures, helping users to understand complex information more intuitively. For example, when answering questions containing flow charts, application forms, product pictures, etc., related pictures can play a supplementary role, allowing users to more accurately obtain the core information of the answer.

[0077] In the above embodiment, when receiving the query question input by the user, the target query text of the query question can be extracted to avoid the input carrier restriction of the query question, and then the target query text can be vectorized to generate a target text vector, and multiple target text blocks can be retrieved from the knowledge base based on the target query text and the target text vector, so that the semantic relevance of the text block can be improved and the retrieval accuracy can be improved; then the target text optimization model can be used to optimize the text of each target text block, and the model can reduce the ambiguity of the text by controlling the output direction of the text, and output a target answer with high compliance and accuracy. In addition, when the target answer contains an image address, the present application can generate a request address based on the image address and the server address. Since the request address includes verification information corresponding to the image address, after initiating an image request to the corresponding server through the request address, the server can perform a security check on the verification information in the request address, and return the image content corresponding to the image address after the verification passes, thereby preventing malicious requests and illegal access, and improving data security; finally, the present application can replace the image address in the target answer with the image content, so that the final target answer can be displayed to the user through the multimodal mode of text and image, thereby improving the intuitiveness and accuracy of the target answer.

[0078] In one embodiment, the process of extracting the target query text of the query question in step S110 may include:

[0079] S111: Determine whether there are other visual information carriers besides text content in the query question.

[0080] S112: If yes, identify the text information of other visual information carriers, and perform key text extraction on the text content and text information to obtain the target query text of the query question.

[0081] S113: If not, directly extract the key text of the text content as the target query text of the query question.

[0082] In this embodiment, when extracting the target query text of the query question, the computer device can first determine whether there are other visual information carriers other than the text content in the query question, and then adopt different extraction methods to extract the target query text according to the determination result. Among them, when there are other visual information carriers other than the text content in the query question, the computer device can identify the text information of the other visual information carriers, and perform key text extraction on the text content and text information to obtain the target query text of the query question; when there are no other visual information carriers other than the text content in the query question, the computer device can directly extract the key text of the text content as the target query text of the query question.

[0083] Among them, other visual information carriers refer to non-text elements that can present information in a visual form, such as pictures and tables, in addition to pure text content. In this application, users can use multiple carriers such as text and pictures to achieve multimodal interaction of intelligent question and answer, so that when users enter query questions, they do not need to fully describe the information in other visual information carriers, thereby reducing the complexity of the interaction process; here, the computer device can obtain additional content or context through other visual information carriers, and then fully understand the user's query question.

[0084] Specifically, if there are other visual information carriers such as pictures and tables in the query question, the computer device can identify the text information of these other visual information carriers. For the content of the picture, the computer device can use OCR (Optical Character Recognition) technology to convert the content in the picture into machine-readable text data; for the content of the table, the computer device can parse the row and column results of the table and convert the data content therein into usable key text. In this way, after the computer device identifies the text information in other visual information carriers, it can combine it with the original text content in the query question and extract the key text in the combined result, so as to ensure that the computer device can fully utilize the information in other visual information carriers such as pictures or tables while obtaining the text content directly input by the user, so as to more comprehensively understand the meaning of the query question and finally generate accurate target query text.

[0085] In one embodiment, Figure 2 A flowchart of a target text block retrieval process provided in an embodiment of the present application; Figure 2 In step S120, the knowledge base pre-stores the text blocks of the plurality of search documents and the search vectors corresponding to each text block; wherein the process of retrieving the plurality of target text blocks from the knowledge base based on the target query text and the target text vectors may include:

[0086] S121: Perform similarity matching between the target query text and each text block in the knowledge base to obtain a first matching result, and perform similarity matching between the target text vector and each search vector in the knowledge base to obtain a second matching result.

[0087] S122: Determine the similarity of each text block according to the first matching result and the second matching result, and sort the text blocks according to the similarities to obtain a sorting result.

[0088] S123: According to a preset similarity threshold and a preset number of selections, a plurality of target text blocks are obtained by screening the text blocks in the sorting results.

[0089] In this embodiment, since the knowledge base pre-stores the text blocks of multiple search documents and the search vectors corresponding to each text block, when the computer device retrieves the target text block, it can first perform similarity matching between the target query text and each text block in the knowledge base to obtain a first matching result, and perform similarity matching between the target text vector and each search vector in the knowledge base to obtain a second matching result, and then determine the similarity of each text block based on the first matching result and the second matching result, and sort the text blocks according to the similarities to obtain a sorting result, and finally, according to a preset similarity threshold and a preset number of selections, screen the text blocks in the sorting result to obtain multiple target text blocks.

[0090] Specifically, the computer device can use the BM25 algorithm to perform text retrieval on the target query text. The BM25 algorithm here refers to an information retrieval algorithm used to evaluate the relevance between the query text and the document. It is an effective document ranking method; its core algorithm lies in its redefinition and calculation of term frequency (TF) and inverse document frequency (IDF), as well as the introduction of a document length normalization factor. The calculation formula of the BM25 algorithm can be shown as follows:

[0091]

[0092] In the formula, Represents the relevance between document D and query text Q; Refers to the inverse document frequency of the i-th word in the query text; Represents the frequency of the i-th word in the query text in document D; and b represent adjustable parameters, where Used to adjust the influence of word frequency, the default value is 1.2; b is used to adjust the impact of document length on word frequency, the default value is 0.75; |D| represents the length of document D; avgdl represents the average length of all documents.

[0093] In addition, the computer device can use cosine similarity to perform vector retrieval on the target text vector. Here, cosine similarity refers to an algorithm that determines the similarity by calculating the ratio of the dot product of two vectors and the product of their moduli. It measures the similarity of the two vectors in direction, not in size. The calculation formula of cosine similarity can be shown as follows:

[0094]

[0095] In the formula, Represents the dot product of vector A and vector B; Represents the product of the modulus of vector A and vector B; n represents the total number of vector components; i represents the index object of the component in the vector; Represents the similarity between vector A and vector B, with a range of [-1, 1], where -1 indicates complete dissimilarity and 1 indicates complete similarity.

[0096] Furthermore, after calculating the first matching result and the second matching result of each text block in the knowledge base, the computer device can determine the first adjustment coefficient of the text search and the second adjustment coefficient of the vector search respectively, and then use the first adjustment coefficient and the second adjustment coefficient to perform weighted summation on the matching results of each text block to obtain the similarity of each text block. Then, the computer device can sort the text blocks according to the similarities to obtain the sorting results, and remove all the text blocks in the sorting results whose similarities are lower than the preset similarity threshold; if the number of text blocks after removal is less than the preset selection number, all the remaining text blocks can be used as target text blocks; if the number of text blocks after removal is more than the preset selection number, text blocks whose number is consistent with the preset selection number can be selected from the remaining text blocks in the order of similarity from high to low as target text blocks.

[0097] In one embodiment, Figure 3 As shown, Figure 3 A schematic diagram of a process of building a knowledge base provided in an embodiment of the present application; Figure 3 In step S120, the process of building the knowledge base may include:

[0098] S1211: Acquire search documents from multiple data sources, and perform text analysis on the search documents to obtain analysis results; the analysis results include text content and other visual information carriers.

[0099] S1212: Extract key texts from other visual information carriers, convert other visual information carriers into image formats, and store them after forming image contents.

[0100] S1213: Generate the image address of other visual information carriers according to the storage ID and key text of the image content.

[0101] S1214: Replace other visual information carriers in the search document with corresponding image addresses, and perform text segmentation on the replaced search document to obtain multiple text blocks.

[0102] S1215: Generate a search vector corresponding to each text block, and construct a knowledge base based on each text block and each search vector.

[0103] In this embodiment, when constructing a knowledge base, the computer device can first obtain a search document from multiple data sources, and perform text analysis on the search document to obtain a parsing result, where the parsing result includes text content and other visual information carriers. Therefore, the computer device can first extract the key text of other visual information carriers, and convert other visual information carriers into image format, store them after forming image content, and then generate the image address of other visual information carriers according to the storage ID and key text of the image content; then, the computer device can replace other visual information carriers in the search document with the corresponding image address, and perform text segmentation on the replaced search document to obtain multiple text blocks, and finally generate a search vector corresponding to each text block, and construct a knowledge base based on each text block and each search vector.

[0104] Specifically, the retrieval documents used in this application to build a knowledge base come from multiple data sources, such as long text content such as TXT, Markdown, DOCX, HTML, PDF, and structured data such as CSV and Excel; when retrieving documents, the computer device can first parse and identify the text content and other visual information carriers, and here the position of each visual information carrier in the retrieval document can be determined by identifying the image URL, the base64 encoded string of the image, or the tag attributes in the document (such as image, table, etc.). Among them, for other visual information carriers such as pictures and tables, the computer device can first extract the key text therein through an OCR tool or a multimodal large model, and then convert it into a picture format for separate storage, and generate a unique storage ID and save it in the local folder of the server to ensure efficient image storage and convenient management. After obtaining the storage ID and key text of other visual text information, the computer device can associate the storage ID with the key text to form a pair of "key text + storage ID" knowledge organization structures; this organizational form can effectively bind the text to the corresponding visual information carrier, which is convenient for subsequent use and display. It should be noted that the storage ID here does not contain the IP address of the server, but points to the path and file name of the stored image, which can increase data security.

[0105] Furthermore, the computer device can replace other visual information carriers in the search document with the corresponding image address, and then perform block processing on the replaced search document to facilitate more accurate indexing and storage. When segmenting, the computer device can split the search document according to the chapters, and then further segment the content of each chapter based on the maximum number of tokens to ensure the sentence integrity and semantic coherence of each text block and avoid information loss or truncation. After the segmentation is completed, the computer device can also vectorize each text block; here the computer device can use the embedding model to convert each text block into a multi-dimensional vector to facilitate efficient storage and retrieval in the knowledge base. Finally, the computer device can build a knowledge base based on the text blocks containing text content and image addresses and the retrieval vector of each text block, so that the knowledge base can simultaneously support text-based and vector-based semantic retrieval and query.

[0106] In one embodiment, the process of optimizing the text of each target text block using the target text optimization model in step S120 to obtain the target answer to the query question may include:

[0107] S124: Determine the large language model prompt word of the query question.

[0108] S125: Input the large language model prompt words and each target text block into the target text optimization model, so that the target text optimization model performs context interpretation on each target text block based on the large language model prompt words, and performs text optimization on each target text block according to the interpretation results to obtain the target answer to the query question.

[0109] In this embodiment, when the computer device performs text optimization on each target text block, it can first determine the large language model prompt words of the query question, and then input the large language model prompt words and each target text block into the target text optimization model, so that the target text optimization model performs contextual interpretation of each target text block based on the large language model prompt words, and performs text optimization on each target text block according to the interpretation results to obtain the target answer to the query question.

[0110] Among them, the large language model prompt words can be used to guide the model to generate expected outputs. It can include information such as the task description that the model needs to perform, the requirements for the output style, the improvement of accuracy, and the guarantee of compliance. It plays a role in setting the direction and quality of the model output, helping to reduce the ambiguity of the model output, improve accuracy, and ensure that the output meets compliance requirements.

[0111] Specifically, the target text optimization model of the present application refers to the LLM model that optimizes the text block based on the large language model prompt words and obtains the target answer. The LLM model is a complex system based on deep learning and natural language processing technology, with up to billions or even hundreds of billions of parameters. By learning a large amount of text data, it can understand and generate human language.

[0112] In one embodiment, the process of generating the verification information in step S130 may include:

[0113] S131: After retrieving the target text block of the query question from the knowledge base, traverse to obtain the image address in each target text block.

[0114] S132: Determine the retrieval time and validity period of the query question, as well as the storage ID in the image address, and use a signature generation algorithm to encrypt and calculate the retrieval time, validity period and storage ID to obtain a first signature.

[0115] S133: Generate verification information of the image address according to the retrieval time, validity period and signature.

[0116] In this embodiment, after the computer device retrieves the target text block of the query question from the knowledge base, it can traverse to obtain the image addresses in each target text block, and then determine the retrieval time and validity period of the query question, as well as the storage ID in the image address, and use a signature generation algorithm to encrypt and calculate the retrieval time, validity period and storage ID to obtain a first signature, and finally generate verification information for the image address based on the retrieval time, validity period and signature.

[0117] Specifically, after the computer device traverses and obtains the image addresses contained in each target text block, it can extract the storage ID in each image address. At the same time, the computer device can also obtain the retrieval time corresponding to the target text block and the pre-set validity period. On this basis, the computer device can use a signature generation algorithm to encrypt and calculate the retrieval time, validity period and storage ID to generate a first signature, and then construct verification information of the image address based on the generated first signature and information such as the retrieval time and validity period. Therefore, the verification information can ensure the security of the data and prevent unauthorized access, and prevent malicious users from obtaining unauthorized resources through forged requests.

[0118] Furthermore, after generating the verification information, the computer device can splice the verification information into the image address, so that the verification information can be carried by the image address and generate an image request to be sent to the server, so that the server can extract the verification information therein for security verification.

[0119] In one embodiment, the security verification process of the verification information in step S140 may include:

[0120] S141: extract the retrieval time, validity period and first signature in the verification information, as well as the storage ID and current time in the image address, and use the signature generation algorithm to encrypt and calculate the retrieval time, validity period and storage ID to obtain the second signature.

[0121] S142: Determine whether the time difference between the retrieval time and the current time is within the validity period and whether the second signature is consistent with the first signature.

[0122] S143: If yes, confirm that the verification is passed.

[0123] S144: If not, confirm that the verification fails.

[0124] In this embodiment, after receiving the image request, the server can extract the image address carrying the verification information in the image request, and then parse the verification information, and extract the retrieval time, validity period and first signature from the parsing result, and then obtain the storage ID and current time in the image address, so that the signature generation algorithm can be used to encrypt and calculate the retrieval time, validity period and storage ID to obtain the second signature. Finally, the server can check whether the time difference between the retrieval time and the current time is within the validity period and whether the second signature is consistent with the first signature; if so, it means that the verification is passed; if not, it means that the verification fails.

[0125] It is understandable that by comparing the second signature with the first signature, the server can check whether the request information has been tampered with; if the two are consistent, it indicates that the request has not been tampered with. In addition, the server also needs to calculate the time difference between the retrieval time and the current time to ensure that the difference is within the preset validity period, thereby confirming that the image request is within the valid time range and has not expired. In this application, only when the second signature is consistent with the first signature and the time difference between the retrieval time and the current time is within the validity period, the server will determine that the verification has passed, and obtain and return the image content based on the image request; if the above conditions are not met, it indicates that the verification has failed, and the server will reject the access request. Through security verification, this application can improve the security of data, and strictly control the request time and validity, providing a strong guarantee for the security and stability of the server.

[0126] In order to better explain the multimodal intelligent question answering method of this application, Figure 4 To further illustrate, schematically, Figure 4 As shown, Figure 4 A flow diagram of a multimodal intelligent question-answering logic provided in an embodiment of the present application.

[0127] Figure 4In the process, the computer device can first build a knowledge base. During the construction process, the document can be processed and the processed image can be stored locally. Then, the document can be divided into multiple document blocks (text blocks) according to the generated image URL (image address), and the vector of each document block is generated by embedding technology to build the knowledge base. Then, the user can ask questions in multiple formats such as text and / or images. After receiving the query question, the computer device can extract the text information of the image when it is in the format of an image, and then use the embedding technology to generate a vector, and perform vector retrieval and full-text retrieval in the knowledge base based on the text information and the vector to obtain the retrieval result; thereafter, the computer device can input the retrieval result into the LLM model so that the LLM model can perform a security check on the verification information carried in the image URL contained in the retrieval result, and obtain the corresponding image content after the verification passes; secondly, the LLM model can also optimize the retrieval result through prompt (large language model prompt word) to generate the final target answer. When the LLM model obtains the image content corresponding to the image URL, the LLM model can automatically identify the image URL in the retrieval result through the front end, and then generate the corresponding image request, and distribute the request through the server Nginx, so that the back end can perform a security check on the verification information carried in the image URL, and obtain the corresponding image content from the local image storage after the check passes and return it to the LLM model, so that the LLM model can finally output the target answer containing text content and image content.

[0128] The multimodal intelligent question-answering device provided in an embodiment of the present application is described below. The multimodal intelligent question-answering device described below and the multimodal intelligent question-answering method described above can be referenced to each other.

[0129] In one embodiment, Figure 5 As shown, Figure 5 A schematic diagram of the structure of a multimodal intelligent question-answering device provided in an embodiment of the present application; the present application also provides a multimodal intelligent question-answering device, including a retrieval data extraction module 210, an answer generation module 220, a request address generation module 230, a picture content acquisition module 240 and an answer update module 250, specifically including the following:

[0130] The retrieval data extraction module 210 is used to receive a query question input by a user, extract a target query text of the query question, and vectorize the target query text to generate a target text vector.

[0131] The answer generation module 220 is used to retrieve multiple target text blocks from the knowledge base based on the target query text and the target text vector, and use the target text optimization model to perform text optimization on each target text block to obtain the target answer to the query question.

[0132] The request address generation module 230 is used to generate a request address based on the image address and the server address when it is detected that the target answer contains the image address; the request address includes verification information corresponding to the image address.

[0133] The picture content acquisition module 240 is used to initiate a picture request to the corresponding server through the request address, so that the server performs a security check on the verification information in the request address, and returns the picture content corresponding to the picture address after the verification passes.

[0134] The answer updating module 250 is used to replace the image address in the target answer with the image content and update the target answer.

[0135] In the above embodiment, when receiving the query question input by the user, the target query text of the query question can be extracted to avoid the input carrier restriction of the query question, and then the target query text can be vectorized to generate a target text vector, and multiple target text blocks can be retrieved from the knowledge base based on the target query text and the target text vector, so that the semantic relevance of the text block can be improved and the retrieval accuracy can be improved; then the target text optimization model can be used to optimize the text of each target text block, and the model can reduce the ambiguity of the text by controlling the output direction of the text, and output a target answer with high compliance and accuracy. In addition, when the target answer contains an image address, the present application can generate a request address based on the image address and the server address. Since the request address includes verification information corresponding to the image address, after initiating an image request to the corresponding server through the request address, the server can perform a security check on the verification information in the request address, and return the image content corresponding to the image address after the verification passes, thereby preventing malicious requests and illegal access, and improving data security; finally, the present application can replace the image address in the target answer with the image content, so that the final target answer can be displayed to the user through the multimodal mode of text and image, thereby improving the intuitiveness and accuracy of the target answer.

[0136] In one embodiment, the retrieval data extraction module 210 may include:

[0137] The carrier determination submodule includes determining whether there are other visual information carriers besides text content in the query question; other visual information carriers include pictures and tables.

[0138] The first text extraction submodule includes, when there are other visual information carriers besides text content in the query question, identifying the text information of other visual information carriers, and performing key text extraction on the text content and text information to obtain the target query text of the query question.

[0139] The second text extraction submodule includes directly extracting key text of the text content as the target query text of the query question when there is no other visual information carrier except the text content in the query question.

[0140] In one embodiment, the knowledge base in the answer generation module 220 pre-stores the text blocks of multiple search documents and the search vectors corresponding to each text block; wherein the answer generation module 220 may include:

[0141] The similarity matching submodule includes performing similarity matching between the target query text and each text block in the knowledge base to obtain a first matching result, and performing similarity matching between the target text vector and each search vector in the knowledge base to obtain a second matching result.

[0142] The text block sorting submodule includes determining the similarity of each text block according to the first matching result and the second matching result, and sorting each text block according to each similarity to obtain a sorting result.

[0143] The text block screening submodule includes screening the text blocks in the sorting results to obtain multiple target text blocks according to a preset similarity threshold and a preset selection number.

[0144] In one embodiment, the answer generation module 220 may further include:

[0145] The text parsing submodule includes obtaining search documents from multiple data sources, and performing text parsing on the search documents to obtain parsing results; the parsing results include text content and other visual information carriers.

[0146] The format conversion storage submodule includes extracting key texts from other visual information carriers, converting other visual information carriers into image formats, and storing them after forming image content.

[0147] The picture address generation submodule includes generating picture addresses of other visual information carriers according to the storage ID and key text of the picture content.

[0148] The text segmentation submodule includes replacing other visual information carriers in the search document with corresponding image addresses, and performing text segmentation on the replaced search document to obtain multiple text blocks.

[0149] The knowledge base construction submodule includes generating a search vector corresponding to each text block, and constructing a knowledge base based on each text block and each search vector.

[0150] In one embodiment, the answer generation module 220 may further include:

[0151] The prompt word determination submodule includes determining the large language model prompt words of the query question.

[0152] The answer query submodule includes inputting the large language model prompt words and each target text block into the target text optimization model, so that the target text optimization model interprets the context of each target text block based on the large language model prompt words, and optimizes the text of each target text block according to the interpretation results to obtain the target answer to the query question.

[0153] In one embodiment, the request address generation module 230 may include:

[0154] The text block traversal submodule includes retrieving the target text block of the query question from the knowledge base and then traversing to obtain the image address in each target text block.

[0155] The first encryption calculation submodule includes determining the retrieval time and validity period of the query question, and the storage ID in the image address, and using a signature generation algorithm to encrypt and calculate the retrieval time, validity period and storage ID to obtain a first signature.

[0156] The verification information submodule includes verification information for generating the image address based on the retrieval time, validity period and signature.

[0157] In one embodiment, the picture content acquisition module 240 may include:

[0158] The second encryption calculation submodule includes extracting the retrieval time, validity period and first signature in the verification information, as well as the storage ID and current time in the image address, and using the signature generation algorithm to encrypt and calculate the retrieval time, validity period and storage ID to obtain the second signature.

[0159] The information verification submodule includes determining whether the time difference between the retrieval time and the current time is within the validity period and whether the second signature is consistent with the first signature.

[0160] The first verification submodule includes confirming that the verification is passed when the time difference between the retrieval time and the current time is within the validity period and the second signature is consistent with the first signature.

[0161] The second verification submodule includes confirming that the verification fails when the time difference between the retrieval time and the current time is not within the validity period or the second signature is not consistent with the first signature.

[0162] In one embodiment, the present application also provides a storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multimodal intelligent question-answering method as described in any of the above embodiments.

[0163] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multimodal intelligent question-answering method as described in any one of the above embodiments.

[0164] Indicatively, Figure 6 As shown, Figure 6 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 6 The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the multimodal intelligent question-answering method of any of the above embodiments.

[0165] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate an operating system based on the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0166] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0167] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0168] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0169] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal intelligent question answering method, characterized in that: The method comprises: Receiving a query question input by a user, extracting a target query text of the query question, and vectorizing the target query text to generate a target text vector; Retrieving multiple target text blocks from a knowledge base based on the target query text and the target text vector, and optimizing each target text block using a target text optimization model to obtain a target answer to the query question; When it is detected that the target answer includes a picture address, generating a request address based on the picture address and the server address; the request address includes verification information corresponding to the picture address; Initiate a picture request to the corresponding server through the request address, so that the server performs a security check on the verification information in the request address, and returns the picture content corresponding to the picture address after the verification passes; The image address in the target answer is replaced with the image content, and the target answer is updated.

2. The multimodal intelligent question-answering method according to claim 1, characterized in that: The step of extracting a target query text of the query question includes: Determine whether there are other visual information carriers besides text content in the query question; the other visual information carriers include pictures and tables: If yes, then identify the text information of the other visual information carriers, and perform key text extraction on the text content and text information to obtain the target query text of the query question; If not, the key text of the text content is directly extracted as the target query text of the query question.

3. The multimodal intelligent question-answering method according to claim 1, characterized in that: The knowledge base pre-stores each text block of a plurality of search documents and a search vector corresponding to each text block; The retrieving a plurality of target text blocks from a knowledge base based on the target query text and the target text vector comprises: Performing similarity matching between the target query text and each text block in the knowledge base to obtain a first matching result, and performing similarity matching between the target text vector and each search vector in the knowledge base to obtain a second matching result; Determining the similarity of each text block according to the first matching result and the second matching result, and sorting the text blocks according to the similarities to obtain a sorting result; According to a preset similarity threshold and a preset number of selections, a plurality of target text blocks are obtained by screening the text blocks in the sorting results.

4. The multimodal intelligent question-answering method according to claim 1 or 3, characterized in that: The process of constructing the knowledge base includes: Acquire a search document from multiple data sources, and perform text analysis on the search document to obtain a analysis result; the analysis result includes text content and other visual information carriers; Extracting key texts from other visual information carriers, converting the other visual information carriers into picture formats, and storing them after forming picture contents; Generate the image address of the other visual information carrier according to the storage ID of the image content and the key text; Replacing other visual information carriers in the search document with corresponding image addresses, and performing text segmentation on the replaced search document to obtain multiple text blocks; A search vector corresponding to each text block is generated, and a knowledge base is constructed based on each text block and each search vector.

5. The multimodal intelligent question-answering method according to claim 1, characterized in that: The target text optimization model is used to optimize the text of each target text block to obtain the target answer to the query question, including: Determine a large language model prompt word for the query question; The large language model prompt words and each target text block are input into the target text optimization model, so that the target text optimization model performs context interpretation on each target text block based on the large language model prompt words, and performs text optimization on each target text block according to the interpretation results to obtain the target answer to the query question.

6. The multimodal intelligent question-answering method according to claim 1, characterized in that: The generation process of the verification information includes: After retrieving the target text block of the query question from the knowledge base, traversing to obtain the image address in each target text block; Determine the retrieval time and validity period of the query question, and the storage ID in the image address, and use a signature generation algorithm to encrypt and calculate the retrieval time, the validity period and the storage ID to obtain a first signature; Verification information of the image address is generated according to the retrieval time, the validity period and the signature.

7. The multimodal intelligent question-answering method according to claim 1, characterized in that: The security verification process of the verification information includes: Extract the retrieval time, validity period and first signature in the verification information, as well as the storage ID and current time in the image address, and use the signature generation algorithm to encrypt and calculate the retrieval time, validity period and storage ID to obtain a second signature; Determine whether the time difference between the retrieval time and the current time is within the validity period and the second signature is consistent with the first signature; If yes, then confirm that the verification has passed; If not, the verification fails.

8. A multimodal intelligent question-answering device, characterized in that: include: A retrieval data extraction module is used to receive a query question input by a user, extract a target query text of the query question, and vectorize the target query text to generate a target text vector; An answer generation module, used to retrieve multiple target text blocks from a knowledge base based on the target query text and the target text vector, and use a target text optimization model to perform text optimization on each target text block to obtain a target answer to the query question; A request address generation module, configured to generate a request address based on the image address and the server address when it is detected that the target answer includes an image address; the request address includes verification information corresponding to the image address; The image content acquisition module is used to initiate an image request to the corresponding server through the request address, so that the server performs a security check on the verification information in the request address, and returns the image content corresponding to the image address after the verification passes; The answer updating module is used to replace the picture address in the target answer with the picture content and update the target answer.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multimodal intelligent question-answering method as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the multimodal intelligent question-answering method as described in any one of claims 1 to 7 are performed.

Images