Multi-modal answer generation method and device

Abstract

The embodiment of the invention provides a multi-modal answer generation method and device.The multi-modal answer generation method comprises the steps that question analysis and answer generation are conducted on a question through a large language model, and at least one sub-question, an associated question, a sub-question answer and an associated question answer are obtained, performing text retrieval in a text database according to the question, each sub-question, the associated question, the sub-question answer and the associated question answer, calculating the semantic similarity of the question and each candidate initial text obtained by the text retrieval through a similarity calculation model, and screening out the initial text of which the semantic similarity is greater than a similarity threshold; the method comprises the following steps of: acquiring multimedia data of a question, performing text merging of initial texts through a large language model to obtain answer texts, and finally splicing the multimedia data and the answer texts according to a position association relationship between the multimedia data associated with the initial texts and the answer texts to obtain multi-modal answer data of the question, so as to realize the generation of the multi-modal answer data of the question.

Description

Technical Field

[0001] This document relates to the field of artificial intelligence technology, and in particular, to a multi-modal answer generation method and device. Background Art

[0002] With the development of Internet technology and artificial intelligence technology, the application of large language models has become more and more common. For example, in the field of intelligent customer service, when a user asks a question, the large language model performs semantic analysis on the user's question and provides an answer, thereby improving the convenience of answering the user's question and also enhancing the user's perception of the corresponding service. However, although large language models have demonstrated powerful capabilities in many aspects, in the process of answer generation, problems such as outdated knowledge and opaque reasoning processes will also be encountered. How to improve the answer generation ability has become the focus of increasing concern for service providers. Summary of the Invention

[0003] One or more embodiments of this specification provide a multi-modal answer generation method, including: inputting a question and a prompt text into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer. Performing text retrieval in a text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain an initial text. Inputting the initial text into the large language model for text merging to obtain an answer text. Determining the positional association relationship between the multimedia data associated with the initial text and the answer text, and splicing the multimedia data and the answer text according to the positional association relationship to obtain the multi-modal answer data of the question.

[0004] One or more embodiments of this specification provide a multi-modal answer generation device, including: a question parsing module configured to input a question and a prompt text into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer. A text retrieval module configured to perform text retrieval in a text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain an initial text. A text merging module configured to input the initial text into the large language model for text merging to obtain an answer text. A multi-modal answer data splicing module configured to determine the positional association relationship between the multimedia data associated with the initial text and the answer text, and splicing the multimedia data and the answer text according to the positional association relationship to obtain the multi-modal answer data of the question.

[0005] One or more embodiments of the present specification provide a multimodal answer generation device, including: a processor; and a memory configured to store computer-executable instructions, which when executed cause the processor to: input a question and prompt text into a large language model for question parsing and answer generation, obtaining at least one sub-question, associated question, sub-question answer, and associated question answer. Perform text retrieval in a text database based on the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain an initial text. Input the initial text into the large language model for text merging to obtain an answer text. Determine the positional association relationship between the multimedia data associated with the initial text and the answer text, and splice the multimedia data and the answer text according to the positional association relationship to obtain the multimodal answer data for the question.

[0006] One or more embodiments of the present specification provide a computer-readable storage medium for storing computer-executable instructions, which when executed implement the following process: input a question and prompt text into a large language model for question parsing and answer generation, obtaining at least one sub-question, associated question, sub-question answer, and associated question answer. Perform text retrieval in a text database based on the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain an initial text. Input the initial text into the large language model for text merging to obtain an answer text. Determine the positional association relationship between the multimedia data associated with the initial text and the answer text, and splice the multimedia data and the answer text according to the positional association relationship to obtain the multimodal answer data for the question. Description of the Drawings

[0007] In order to more clearly illustrate the technical solutions in one or more embodiments of the present specification or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. Figure 1 It is a schematic diagram of the implementation environment of a multimodal answer generation method provided by one or more embodiments of the present specification. Figure 2 It is a processing flow chart of a multimodal answer generation method provided by one or more embodiments of the present specification. Figure 3 It is a schematic diagram of the multimodal answer generation process applied to a multimodal scenario provided by one or more embodiments of the present specification. Figure 4A processing flow chart of a multi-modal answer generation method provided for one or more embodiments of this specification; Figure 5 A schematic diagram of an embodiment of a multi-modal answer generation device provided for one or more embodiments of this specification; Figure 6 A structural schematic diagram of a multi-modal answer generation device provided for one or more embodiments of this specification. Detailed implementation manners

[0008] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will clearly and completely describe the technical solutions in one or more embodiments of this specification in conjunction with the accompanying drawings in one or more embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this document.

[0009] The multi-modal answer generation method provided by one or more embodiments of this specification is applicable to the implementation environment of multi-modal answer generation. Referring to Figure 1 , this implementation environment at least includes: A question parsing module 101, a text retrieval module 102, a merging module 103, a position association relationship determination module 104, a multi-modal answer data splicing module 105, a large language model 106, and a text database 107; Among them, the question parsing module 101 is used to perform question parsing and answer generation using the large language model 106 to obtain at least one sub-question, associated question, sub-question answer, and associated question answer; The text retrieval module 102 is used to perform text retrieval in the text database 107 according to the question, each sub-question, associated question, sub-question answer, and associated question answer to obtain the initial text; The merging module 103 is used to perform text merging on the initial text using the large language model 106 to obtain the answer text; The position association relationship determination module 104 is used to determine the position association relationship between the initial text-associated multimedia data and the answer text; The multi-modal answer data splicing module 105 is used to splice the multimedia data and the answer text according to the position association relationship to obtain the multi-modal answer data of the question; The large language model 106 is used to perform question parsing and answer generation on the question, and perform text merging on the initial text; the model 106 can be a large language model (LLM, Large Language Models); The text database 107 is used to store the initial text that composes the multimedia data associated with the multimedia data.

[0010] In this implementation environment, during the process of generating multi-modal answer data, first, the question parsing module 101 uses the large language model 106 to perform question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer. Then, the text retrieval module 102 performs text retrieval in the text database 107 based on the question, each sub-question, associated question, sub-question answer, and associated question answer to obtain the initial text. Next, the merging module 103 uses the large language model 104 to perform text merging to obtain the answer text. Then, the position association relationship determination module 104 determines the position association relationship between the multimedia data associated with the initial text and the answer text. Finally, the multi-modal answer data splicing module 105 splices the multimedia data and the answer text according to the position association relationship to obtain the multi-modal answer data for the question. In this way, the generation of the multi-modal answer data for the question is realized.

[0011] One or more embodiments of a multi-modal answer generation method provided in this specification are as follows: Refer to Figure 2 , the multi-modal answer generation method provided in this embodiment, the method specifically includes steps S202 to S208.

[0012] Step S202, input the question and the prompt text into the large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer.

[0013] The large language model in this embodiment includes an LLM (Large Language Model); in addition, the large language model described in this embodiment can also be a pre-trained natural language model. The large language model can adopt a foundation model, and can also adopt a pre-trained model. The specific architecture of the large language model can be a neural network architecture with a large number of parameters, a Transform architecture, or other architectures. During the specific execution process, the large language model can directly adopt the foundation model or the pre-trained model, or can fine-tune (Supervised Fine-Tuning, SFT) the foundation model or the pre-trained model for the question parsing and answer generation task on the basis of the foundation model or the pre-trained model, and then a large language model capable of performing question parsing and answer generation can be obtained.

[0014] In specific implementation, after obtaining the question submitted by the user, the question and the prompt text are input into the large language model for question parsing and answer generation, obtaining at least one sub-question, associated question, sub-question answer, and associated question answer. Optionally, the prompt text includes the prompt text for the large language model to perform question parsing and generate answers based on the parsed questions; the question parsing includes question splitting and / or question association. In addition, the question parsing can also be replaced by question rewriting.

[0015] In the specific execution process, during the process of the large language model performing question parsing, it can first split the question to obtain at least one sub-question, and then perform an association process on the historical question and the question to obtain an associated question. Further, during the process of answer generation, sub-question answers are generated according to each sub-question, and associated question answers are generated according to the associated question.

[0016] Specifically, the large language model first performs question parsing according to the question parsing keywords in the prompt text to obtain at least one sub-question and an associated question, and then generates answers for each sub-question and associated question according to the answer generation keywords in the prompt text, obtaining sub-question answers and associated question answers for each sub-question.

[0017] During this process, the prompt text can be: split multiple sub-questions that may be involved in the question; associate the historical question of the historical conversation with the current question; generate answers for the obtained questions respectively. In addition, in order to improve the effectiveness of the large language model's question parsing and answer generation, the prompt text can also include parsing requirement data and / or answer requirement data. For example, the prompt text is: (1) Split multiple sub-questions that may be involved in the question. During the splitting process, first judge whether the question needs to be split. If you think the question contains only one question, then there is no need to split. Otherwise, it needs to be split. If it needs to be split, try to use the words used in the question to generate sub-questions as much as possible to reduce the diversity of question splitting; (2) Associate the historical question of the historical conversation with this question to obtain an associated question that completes the information; (3) Generate answers for the obtained questions respectively, ensuring that the answers are reasonable, can be understood and responded to, and are generated according to common sense.

[0018] In addition to problem splitting and / or problem association, in order to further improve the effectiveness of problem parsing and enhance the comprehensiveness of the obtained problem parsing results, problem parsing may further include problem correction and / or problem extension; in an optional implementation manner provided in this embodiment, problem parsing further includes obtaining a corrected problem by correcting the problem and / or obtaining an extended problem by extending the problem; correspondingly, answer generation may further include generating a corrected problem answer and / or an extended problem answer according to the corrected problem and / or the extended problem. Among them, problem correction may be to correct the content of the problem, including completing the content of the problem and / or correcting typos; problem extension may be to rewrite the problem into a more detailed version.

[0019] It should be noted that inputting the problem and the prompt text into the large language model for problem parsing and answer generation, and obtaining at least one sub-problem, associated problem, sub-problem answer, and associated problem answer can be replaced by inputting the problem and the prompt text into the large language model for problem parsing and / or answer generation, and obtaining a processing result; among them, the processing result may include one or more of the above at least one sub-problem, associated problem, sub-problem answer, associated problem answer, corrected problem, corrected problem answer, extended problem, and extended problem answer, which is not limited in this embodiment; It should also be noted that when inputting the problem and the prompt text into the large language model for problem parsing and answer generation, and obtaining at least one sub-problem, associated problem, sub-problem answer, and associated problem answer is replaced by inputting the problem and the prompt text into the large language model for problem parsing and / or answer generation, and obtaining a processing result or obtaining one or more of the above, the following step S204 can be replaced by performing text retrieval in the text database according to the processing result or one or more of the above, obtaining an initial text, and forming a new implementation manner with step S206 and step S208.

[0020] Step S204, perform text retrieval in the text database according to the problem, each sub-problem, the associated problem, the sub-problem answer, and the associated problem answer, and obtain an initial text.

[0021] The initial text in this embodiment includes pre-obtained text category / format data; in this embodiment, the text of the text category and the multimedia data associated with the text form multi-modal data. The multi-modal data in this embodiment includes multi-modal data composed of at least two categories of data; for example, the multi-modal data includes text and images; among them, the text is the text category, and the image is the multimedia category. Optionally, the multi-modal data includes text and multimedia data. Among them, the multimedia data may be images, videos, animations, interactive content, and / or composite multimedia.

[0022] In specific implementation, the question and the prompt text are input into a large language model for question parsing and answer generation to obtain at least one sub-question. After associating the related question, sub-question answers, and related question answers, text retrieval is performed in the text database based on the question, each sub-question, related question, sub-question answers, and related question answers to obtain the initial text.

[0023] In the specific execution process, the similarity between the question, each sub-question, related question, sub-question answers, and / or related question answers and each text in the text database can be calculated respectively, and the initial text with a similarity greater than other texts and / or greater than the similarity threshold is screened out.

[0024] In specific implementation, in order to improve the efficiency of text retrieval, vector data of the text can be stored in the text database; that is, the text database can be a vector database; the vector data in the vector database can be obtained by encoding the text. In the process of performing text retrieval in the text database according to the question, each sub-question, related question, sub-question answers, and related question answers, the question, each sub-question, related question, sub-question answers, and / or related question answers can be encoded respectively to obtain the corresponding question vector data and / or answer vector data, and then vector data retrieval is performed in the vector database according to the question vector data and / or answer vector data to obtain the initial text corresponding to the vector data.

[0025] In the process of encoding the question, each sub-question, related question, sub-question answers, and / or related question answers respectively, the question, each sub-question, related question, sub-question answers, and / or related question answers can be input into an encoder for encoding to obtain the corresponding question vector data and / or answer vector data; among them, the encoder can be a Bi-encoder. Furthermore, in the process of performing text retrieval or vector text retrieval in the text database or vector database according to the question vector data and / or answer vector data to obtain the initial text, the similarity between the question vector data and / or answer vector data and each vector data in the vector database can be calculated, and the target vector data with a similarity greater than other vector data or greater than the similarity threshold is screened out, and the text corresponding to the target vector data is obtained as the initial text.

[0026] Taking a sub-question as an example, in the process of performing text retrieval in the text database according to the sub-question to obtain the initial text, the sub-question is first input into an encoder for encoding to obtain the question encoding data, and then the cosine similarity between the question encoding data and each vector data in the vector database is calculated respectively, and the target vector data with a cosine similarity greater than other vector data is screened out, and the initial text corresponding to the target vector data is obtained.

[0027] It should be noted that when performing text retrieval in the text database according to a sub-question, the number of initial texts obtained can be one or multiple. Correspondingly, when performing text retrieval in the text database according to the question, associated questions, sub-question answers, and / or associated question answers, the number of initial texts obtained can be one or multiple. The number of final obtained initial texts can be configured according to the actual scenario, and this embodiment does not make any limitations here.

[0028] In practical applications, since text retrieval is performed in the text database based on different input parameters, there may be duplicate content among the multiple initial texts obtained. To improve the efficiency and convenience of subsequent processing and avoid duplicate processing, in this embodiment, after performing text retrieval to obtain multiple initial texts, the multiple initial texts can be de-duplicated.

[0029] Specifically, to improve the matching degree between the obtained initial texts and the question, during the process of performing text retrieval in the text database according to the question, each sub-question, associated questions, sub-question answers, and associated question answers to obtain the initial texts, multiple candidate initial texts can be first obtained by performing text retrieval in the text database according to the question, each sub-question, associated questions, sub-question answers, and associated question answers, and then the initial texts whose similarity to the question meets the similarity condition can be screened out from the multiple candidate initial texts; during the process of specifically performing text retrieval in the text database according to the question, each sub-question, associated questions, sub-question answers, and associated question answers to obtain multiple candidate initial texts, text retrieval can be performed through the text retrieval methods provided by the above relevant content to obtain multiple candidate initial texts; the similarity condition includes that the semantic similarity is greater than the similarity threshold, and / or the number of initial texts is less than the number threshold.

[0030] On this basis, the initial texts whose similarity to the question meets the similarity condition are screened out from the multiple candidate initial texts; during the process of screening the initial texts whose similarity to the question meets the similarity condition from the multiple candidate initial texts, the semantic similarity between the question and each candidate initial text can be first calculated, and then the candidate initial texts with semantic similarity greater than the similarity threshold are determined as the initial texts; in an optional implementation manner provided by this embodiment, during the process of screening the initial texts whose similarity to the question meets the similarity condition from the multiple candidate initial texts, the following operations are performed: The question and each candidate initial text are input into the similarity calculation model for similarity calculation to obtain the semantic similarity between the question and each candidate initial text; The candidate initial texts with semantic similarity greater than the similarity threshold are determined as the initial texts.

[0031] Optionally, the similarity calculation model can be the Cross encoder.

[0032] Specifically, for a question and any candidate initial text, the question and the candidate initial text can be concatenated through an identifier to obtain a concatenated text, and the concatenated text is input into a similarity calculation model for similarity calculation to obtain the semantic similarity between the question and the candidate initial text.

[0033] For example, the question and each candidate initial text are concatenated together through a SEP identifier to obtain a concatenated text, and the concatenated text is input into a Cross encoder for similarity calculation to obtain the semantic similarity between the question and each candidate initial text. The candidate initial text with a semantic similarity greater than the similarity threshold to the question is determined as the initial text.

[0034] It should be noted that the process of obtaining the initial text by performing text retrieval in the text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer can also be to perform text retrieval in the text database according to each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain multiple candidate initial texts, perform a deduplication process on the multiple candidate initial texts, and then screen out the initial text that meets the similarity condition with the question from the deduplicated candidate initial texts. That is, step S204 can be replaced by performing text retrieval in the text database according to each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain the initial text, and forming a new implementation method with one or more other processing steps provided in this embodiment.

[0035] In this embodiment, the text database contains multiple texts, and the texts contained in the text database are associated with multimedia data. The text and the associated multimedia data form multimodal data; by pre-constructing the text database, text retrieval in the text database is facilitated during the process of answer generation, and the efficiency of answer generation is also improved. During the specific execution process, multiple multimodal data can be obtained through document preprocessing and partitioning processing, and then the text database can be constructed according to the texts in each multimodal data; the generation process of the multiple multimodal data is specifically described below.

[0036] (1) Preprocessing In an alternative implementation provided by this embodiment, during the preprocessing process, the document multimedia data can be locally stored to obtain a local storage identifier, and the document multimedia data in the document is replaced with the local storage identifier to obtain document data including the document text and the local storage identifier. In this way, the layout of the text and multimedia data of the document is retained, that is, the document data retaining the relative position information of the document text and multimedia data is obtained, so as to associate the multimedia data with the multi-modal data subsequently. In this embodiment, a multi-modal data can be a multi-modal data block composed of text and multimedia data. For example, a text-image block composed of text and images.

[0037] In the specific execution process of preprocessing, the document text and the document multimedia data in the document can be preprocessed separately. The following separately describes the preprocessing of the document text and the preprocessing of the document multimedia data.

[0038] a. Document text preprocessing In the process of preprocessing the initial document text in the document, the initial document text is text-cleaned to obtain the document text. In this way, the document text with noise text removed is obtained.

[0039] b. Multimedia data preprocessing In the process of preprocessing the document multimedia data in the document, the document multimedia data can be first locally stored to obtain a local storage identifier. To ensure that the obtained document data is all text and to make the document data retain the relative position information of the document text and multimedia data, the document multimedia data in the document is replaced with the local storage identifier.

[0040] In this way, through the document text preprocessing and the multimedia data preprocessing, document data including the document text and the local storage identifier of the document multimedia data is obtained. The local storage identifier includes the local storage address.

[0041] In addition, to improve the effectiveness of the locally stored multimedia data, if the multimedia data contains text, the multimedia text in the multimedia data is identified and used as the title of the locally stored multimedia data. For example, in the process of identifying the image text in an image, the OCR (Optical Character Recognition) technology can be used to identify the image text in the image. The document data in this embodiment can be a markdown string.

[0042] It should be noted that the document multimedia data in the document can also be replaced with the corresponding multimedia text to obtain document data including the document text and the multimedia text.

[0043] (2)Partitioning process In specific implementation, after obtaining the document data, the document data is partitioned to obtain multiple multimodal data. In an optional implementation manner provided in this embodiment, in the process of partitioning the document data to obtain multiple multimodal data, the following operations are performed: Read the multimedia text of the document multimedia data according to the local storage identifier; Extract features from the document text and the multimedia text in the document data to obtain a text feature matrix; Calculate a similarity matrix according to the text feature matrix and a preset constraint matrix, and partition the document data according to the similarity matrix to obtain multiple multimodal data.

[0044] In the specific execution process, the TF-IDF method (Term Frequency-Inverse Document Frequency) can be used to extract features from the document text and the multimedia text to obtain a TF-IDF matrix, calculate the similarity of each text feature in the TF-IDF matrix using the cosine similarity formula, obtain an initial similarity matrix as the text feature matrix, and then subtract the initial similarity matrix from the preset constraint matrix to obtain the similarity matrix. Partition the document data according to the similarity matrix to obtain multiple multimodal data. In the process of partitioning the document data according to the similarity matrix to obtain multiple multimodal data, the sum of similarities in the virtual data can be maximized as the objective, and the dynamic programming algorithm is used to partition the document data to obtain multiple multimodal data; in this process, to improve the efficiency of dynamic programming, the dynamic programming can be accelerated through windowing and matrix operations.

[0045] Specifically, calculate the semantic similarity between texts through the TF-IDF matrix and the cosine similarity to obtain an initial similarity matrix, adjust the initial similarity matrix according to the preset constraint matrix to obtain the similarity matrix, calculate the document partitioning scheme using the similarity matrix, and partition the document data according to the document partitioning scheme to obtain multiple multimodal data; in this way, both semantic similarity and the physical position relationship between texts are considered, realizing efficient and effective partitioning of document data. Optionally, the document partitioning scheme is obtained using the dynamic programming algorithm.

[0046] It should be noted that for the document multimedia data with multimedia text, the above method can be used to divide it into multi-modal data. For the document multimedia data with empty multimedia text, if the document multimedia data is located between two multi-modal data, the document multimedia data is respectively associated with the text input index calculation model in the multi-modal data to calculate the association index, and the association index between the document multimedia data and each multi-modal data is obtained; the document multimedia data is added to the multi-modal data with a larger association index. Among them, the index calculation model can be a correlation score calculation model or a multi-modal large model, and the multi-modal large model can be obtained by fine-tuning the open-source multi-modal large model InternVL2-8B. Among them, the association index can be a correlation score.

[0047] On the basis of obtaining multiple multi-modal data, in this embodiment, in order to improve the convenience and efficiency of subsequent text retrieval, in this embodiment, a multi-modal data index and a vector data index obtained by encoding the text in the multi-modal data are constructed, that is, a mapping relationship between the multi-modal data and the corresponding vector data is established, and a text database is constructed according to the vector data of the text in each multi-modal data; In this process, the text in the multi-modal data can be input into the encoder for encoding to obtain vector data, and a vector data index and a multi-modal data index corresponding to the multi-modal data are constructed. In order to improve the effectiveness of the obtained vector data of the text, the text can be preprocessed first, and the preprocessed text is input into the encoder for encoding to obtain vector data. Among them, the preprocessing of the text includes removing the links and format symbols in the text.

[0048] In addition to retrieving text in the text database according to the question, each sub-question, associated question, sub-question answer, and associated question answer to obtain the initial text, data can also be retrieved in the multi-modal database according to the question, each sub-question, associated question, sub-question answer, and associated question answer to obtain multi-modal data. Correspondingly, the following process of inputting the initial text into the large language model for text merging to obtain the answer text, determining the positional association relationship between the multimedia data associated with the initial text and the answer text, and splicing the multimedia data and the answer text according to the positional association relationship to obtain the multi-modal answer data of the question can also be replaced by inputting the initial text in the multi-modal data into the large language model for text merging to obtain the answer text, determining the positional association relationship between the multimedia data associated with the multi-modal data and the answer text, and splicing the multimedia data and the answer text according to the positional association relationship to obtain the multi-modal answer data of the question.

[0049] It should be noted that if it is replaced to obtain multimodal data, the specific implementation process is similar to the above process, and the above content can be modified accordingly. This embodiment will not be elaborated here. For example, after retrieving data in the multimodal database according to the question, each sub-question, related questions, sub-question answers, and related question answers, and obtaining multiple candidate multimodal data, multimodal data that meets the similarity condition with the question can be screened out from the multiple candidate multimodal data; in the process of screening multimodal data that meets the similarity condition with the question from the multiple candidate multimodal data, the semantic similarity between the question and the text in each candidate multimodal data can be calculated first, and then the candidate multimodal data with a semantic similarity greater than the similarity threshold can be determined as multimodal data; in an optional implementation manner provided in this embodiment, in the process of screening multimodal data that meets the similarity condition with the question from the multiple candidate multimodal data, the following operations are performed: input the question and the text in each candidate multimodal data into a similarity calculation model to calculate the similarity, and obtain the semantic similarity between the question and each candidate multimodal data; determine the candidate multimodal data with a semantic similarity greater than the similarity threshold as multimodal data.

[0050] Step S206, input the initial text into a large language model for text merging to obtain an answer text.

[0051] In specific implementation, based on the obtained initial text, the initial text is merged to obtain an answer text; in the process of merging the initial text to obtain an answer text, the initial text can be input into a large language model for text merging to obtain an answer text. The initial text includes the text in the multimodal data.

[0052] In the specific execution process, in order to improve the effectiveness of the answer text in answering the question, the question and the initial text can also be input into a large language model for text merging according to the question to obtain an answer text.

[0053] In specific implementation, in the case of obtaining multiple initial texts, in order to avoid directly inputting the question and all initial texts into a large language model, which may cause the amount of data processed by the large language model to be too large during the text merging process according to the question and affect the text merging efficiency, the question and each initial text can be input into a large language model for text merging first to obtain multiple initial answer texts; then the question and the multiple initial answer texts are input into a large language model for text merging to obtain an answer text.

[0054] In addition, the problem, the merged prompt text, and the initial text in the multimodal data can be input into the large language model for text merging to obtain the answer text. Among them, the merged prompt text can be used to instruct the large language model to perform text merging according to the problem. It should be noted that the large language model for text merging can be the large language model for problem parsing and answer generation mentioned above, or a large language model capable of performing text merging obtained by fine-tuning (Supervised Fine-Tuning, SFT) the base model or the pre-trained model for the text merging task.

[0055] To improve the effectiveness of the answer text obtained by the large language model for text merging, the merged prompt text in this embodiment may include merging requirement data. For example, the merged prompt text is: First, perform a preliminary detection. During the preliminary detection, ensure the authenticity of the merging, and ensure not to generate any text other than the provided initial text, and do not discard information related to the problem. Specifically, first determine the relevance between the problem and the initial text. If they are not relevant, return an empty result. If they are relevant, determine whether the initial text can answer the problem. If it cannot answer, return an empty result. If it can answer, focus on the relevant initial text and ignore the irrelevant initial text. Then, perform text merging or text summarization based on the problem and the relevant initial text. During this process, ensure the coherence of the context semantics.

[0056] Step S208: Determine the positional association relationship between the multimedia data associated with the initial text and the answer text, and splice the multimedia data and the answer text according to the positional association relationship to obtain the multimodal answer data for the problem.

[0057] In specific implementation, to obtain the multimodal answer data, based on the obtained answer text, splice the answer text and the multimedia data associated with the obtained initial text to obtain the multimodal answer data for the problem. During this process, to improve the perception degree of the answer text and the multimedia data in the multimodal answer data, first determine the multimedia data associated with the initial text, that is, the positional association relationship between the multimedia data in the multimodal data to which the initial text belongs and the answer text, and then splice the multimedia data and the answer text according to the positional association relationship to obtain the answer data for the problem. In this way, the multimodal answer data including the multimedia data and the answer text is obtained.

[0058] In the process of specifically determining the positional association relationship between the multimedia data and the answer text, a positional association relationship between one or two sub-texts in the multimedia data and the answer text with an association index greater than other sub-texts can be established. Among them, the association index can be a correlation score or a correlation rating.

[0059] In an alternative implementation provided by this embodiment, the process of determining the positional association relationship between the multimedia data associated with the initial text and the answer text is implemented in the following manner: Perform association index calculation on the multimedia data and each sub-text in the answer text using an index calculation model to obtain the association index between the multimedia data and each sub-text; Determine the target sub-text among multiple sub-texts based on the association index, and establish the positional association relationship between the multimedia data and the target sub-text.

[0060] Specifically, each text paragraph or each text sentence in the answer text can be used as a sub-text. The association index calculation is performed on the multimedia data and each sub-text using the index calculation model respectively to obtain the association index between the multimedia data and each sub-text. Then, two adjacent sub-texts with an association index greater than that of other sub-texts with respect to the multimedia data among multiple sub-texts are determined as the target sub-texts, and the positional association relationship between the multimedia data and the two target sub-texts is established.

[0061] On the basis of establishing the positional association relationship between the multimedia data and the two target sub-texts, when splicing the multimedia data and the answer text according to the positional association relationship to obtain the answer data, the multimedia data is spliced in the middle of the two associated target sub-texts according to the positional association relationship to obtain the multimedia answer data for the question.

[0062] It should be noted that the above description illustrates the determination of the positional association relationship and the splicing with the answer text using one multimedia data as an example. An initial text can be associated with multiple multimedia data, and multiple initial texts will also contain multiple multimedia data. For each multimedia data, the above method is used to determine the positional association relationship and splice it with the answer text; It should also be noted that if the target sub-texts of at least two multimedia data are the same, then the at least two multimedia data are sorted in descending order according to the association index between each multimedia data and the first sub-text (the previous sub-text) in the target sub-text to obtain a multimedia data list, and the positional association relationship between the multimedia data list and the target sub-text is established; That is: after performing association index calculation on each multimedia data and each sub-text input index calculation model respectively to obtain the association index between each multimedia data and each sub-text, determine the target sub-text of each multimedia data and establish the positional association relationship between each multimedia data and the corresponding target sub-text; if the target sub-texts of at least two multimedia data are the same, then sort the at least two multimedia data in descending order according to the association index with the first sub-text to obtain a multimedia data list, and establish the positional association relationship between the multimedia data list and the target sub-text.

[0063] In order to further enhance the perception of multimodal answer data, in an optional implementation provided in this embodiment, after obtaining the multimodal answer data, the multimodal answer data is marked based on the data identifier of the multimodal data to which the initial text belongs; in this process, each sub-text and the initial text in the multimodal answer data can be input into a similarity calculation model to calculate the similarity, obtaining the text semantic similarity. The data identifier of the multimodal data to which the initial text with a text semantic similarity greater than the similarity threshold belongs is marked as the subscript of the corresponding sub-text, so as to perceive the source of the sub-text through the multimodal answer data and enhance the trust in the multimodal answer data.

[0064] In summary, for one or more multimodal answer generation methods provided in this embodiment, during the process of generating multimodal answers to questions, first, the question and the prompt text are input into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer. Then, data retrieval is performed in the dataset based on each sub-question, associated question, sub-question answer, and associated question answer to obtain candidate multimodal data including the initial text and multimedia data; thus, by deriving sub-questions, associated questions, sub-question answers, and associated question answers from the question, and further performing data retrieval based on the sub-questions, associated questions, sub-question answers, and associated question answers, the comprehensiveness and diversity of the obtained multimodal data are improved; since answers are generated for the question, therefore, the semantic similarity between the question and the initial text in each candidate multimodal data is calculated, and the multimodal data with a semantic similarity greater than the similarity threshold is determined, thus retaining the multimodal data with a higher semantic similarity to the question and improving the effectiveness of the obtained multimodal data; on this basis, the question and the initial text in the multimodal data are input into a large language model to perform text merging of the initial text according to the text, obtaining an answer text. According to the association index between the multimedia data in the multimodal data and the sub-text in the answer text, the positional association relationship between the multimedia data and the answer text is determined, and the multimedia data and the answer text are spliced according to the positional association relationship to obtain the answer data to the question, thus improving the matching degree between the multimedia data and the answer text in the generated answer data, obtaining multimodal answer data including the answer text and multimedia data, and improving the comprehensiveness and effectiveness of the answer data.

[0065] The following takes the application of a multimodal answer generation method provided in this embodiment in a multimodal scenario as an example, in combination with Figure 3 and Figure 4 , to further illustrate the multimodal answer generation method provided in this embodiment. As Figure 4 shown, the multimodal answer generation method applied to a multimodal scenario specifically includes the following steps.

[0066] As Figure 3As shown in the figure, the process of answer generation is implemented through a question parsing module, a retrieval module, a filtering module, and an answer generation module. Specifically, the question parsing module first performs question splitting, question association, question correction, and / or question extension based on the question, obtains at least one sub-question, associated question, corrected question, and / or extended question, and then generates answers to obtain sub-question answers, associated question answers, corrected question answers, and / or extended question answers. Here, the sub-questions, associated questions, corrected questions, and / or extended questions, as well as the sub-question answers, associated question answers, corrected question answers, and / or extended question answers can be used as processing results. On this basis, the retrieval module performs text retrieval in the text database according to the processing results, obtains candidate initial texts and removes duplicates, and then the filtering module filters the candidate initial texts according to the semantic similarity between the question and the de-duplicated candidate initial texts to obtain the initial text. Finally, the answer generation module first inputs the initial text into a large language model for text merging to obtain the answer text, and then splices the image associated with the initial text with the answer text to obtain the multi-modal answer data of the question.

[0067] In the process of constructing the text database, for documents in formats such as html, pdf, and word, preprocessing is first performed. During the preprocessing process, image dumping and text cleaning are carried out to obtain a markdown string, and then chunking is performed according to the markdown string. Specifically, during the chunking process, chunking is performed according to a preset length and semantic similarity, and text and image association is carried out to obtain multi-modal data, and then indexing is established. Specifically, feature encoding is performed on the text in each multi-modal data to obtain vector data, and a vector database is constructed based on the vector data and a vector data index is established; so as to determine the multi-modal data corresponding to the target vector data after obtaining the target vector data from the vector database according to the vector data index and the multi-modal data index.

[0068] Step S402: Input the question and the prompt text into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer.

[0069] Optionally, step S402 can be replaced by parsing the question and generating answers to obtain at least one sub-question, associated question, sub-question answer, and associated question answer, and forming a new implementation manner with other processing steps provided in this embodiment.

[0070] Step S404: Perform data retrieval in the vector database according to each sub-question, associated question, sub-question answer, and associated question answer to obtain candidate vector data, and read the candidate initial text in the multi-modal data corresponding to the candidate vector data.

[0071] Step S406: Input the question and each candidate initial text into a cross-encoder for similarity calculation to obtain the semantic similarity between the question and each candidate initial text.

[0072] Step S408: Determine the candidate initial texts with semantic similarity greater than the similarity threshold as the initial texts.

[0073] Step S410: Input the question and the initial texts into a large language model for text merging to obtain the answer text.

[0074] Optionally, Step S410 can be replaced with: perform text merging on the initial texts according to the question to obtain the answer text, and form a new implementation manner with other processing steps provided in this embodiment.

[0075] Step S412: Input the image in the multimodal data and each sub-text in the answer text into a multimodal large model for correlation score calculation to obtain the correlation scores between the image and each sub-text.

[0076] Optionally, Step S412 can be replaced with: input the image associated with the initial text and each sub-text in the answer text into a multimodal large model for correlation score calculation to obtain the correlation scores between the image and each sub-text, and form a new implementation manner with other processing steps provided in this embodiment.

[0077] Step S414: Determine two adjacent target sub-texts among the multiple sub-texts of the answer text according to the correlation scores.

[0078] Step S416: Establish a position association relationship between the image and the middle position of the two adjacent target sub-texts.

[0079] Step S418: Stitch the image and the answer text according to the position association relationship to obtain the multimodal answer data of the question.

[0080] In addition, Step S404 to Step S418 can be replaced with: perform data retrieval in the multimodal database according to each sub-question, associated question, sub-question answer, and associated question answer to obtain the initial text and the image; input the question and the initial text into a large language model for text merging to obtain the answer text, input the image and each sub-text in the answer text into a multimodal large model for correlation score calculation to obtain the correlation scores between the image and each sub-text, determine the target sub-texts among the multiple sub-texts of the answer text according to the correlation scores, establish a position association relationship between the image and the target sub-texts, stitch the image and the answer text according to the position association relationship to obtain the multimodal answer data of the question, and form a new implementation manner with other processing steps provided in this embodiment.

[0081] It should be noted that any one step or any combination of steps from step S402 to step S418 can be combined with any one step or any combination of steps from the above-mentioned step S202 to step S208 according to the needs of implementation and deployment to form a new implementation method; in addition, according to the actual deployment needs, any one or any combination of technical features in step S402 to step S418 can be selected and combined with any one or more technical features provided in the above-mentioned step S202 to step S208 to form a new implementation method; or, any one or any combination of technical features in step S402 to step S418 can also be replaced according to the actual deployment needs by any one or more technical feature combinations provided in the above-mentioned step S202 to step S208 to form a new implementation method, which will not be elaborated here one by one.

[0082] An embodiment of a multimodal answer generation device provided in this specification is as follows: In the above embodiment, a multimodal answer generation method is provided. Correspondingly, a multimodal answer generation device is also provided, which will be described below with reference to the accompanying drawings.

[0083] Refer to Figure 5 , which shows a schematic diagram of an embodiment of a multimodal answer generation device provided in this embodiment.

[0084] Since the device embodiment corresponds to the method embodiment, the description is relatively simple. For the relevant parts, please refer to the corresponding description of the method embodiment provided above. The device embodiments described below are only illustrative.

[0085] This embodiment provides a multimodal answer generation device, and the device includes: A question parsing module 502, configured to input a question and a prompt text into a large language model for question parsing and answer generation, and obtain at least one sub-question, an associated question, a sub-question answer, and an associated question answer; A text retrieval module 504, configured to perform text retrieval in a text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer, and obtain an initial text; A text merging module 506, configured to input the initial text into a large language model for text merging to obtain an answer text; A multimodal answer data splicing module 508, configured to determine the position association relationship between the multimedia data associated with the initial text and the answer text, and splice the multimedia data and the answer text according to the position association relationship to obtain the multimodal answer data of the question.

[0086] Optionally, the text retrieval module 504 includes: A retrieval sub-module, configured to perform text retrieval in the text database according to the problem, the sub-problems, the associated problems, the sub-problem answers, and the associated problem answers, to obtain a plurality of candidate initial texts; A filtering sub-module, configured to screen out the initial texts that meet the similarity condition with the problem from the plurality of candidate initial texts.

[0087] Optionally, the filtering sub-module includes: A similarity calculation unit, configured to input the problem and each candidate initial text into a similarity calculation model for similarity calculation, to obtain the semantic similarity between the problem and each candidate initial text; An initial text determination unit, configured to determine the candidate initial texts with a semantic similarity greater than the similarity threshold as the initial texts.

[0088] An embodiment of a multi-modal answer generation device provided in this specification is as follows: Corresponding to the above-described multi-modal answer generation method, based on the same technical concept, one or more embodiments of this specification also provide a multi-modal answer generation device, which is used to execute the above-provided multi-modal answer generation method. Figure 6 It is a schematic structural diagram of a multi-modal answer generation device provided by one or more embodiments of this specification.

[0089] A multi-modal answer generation device provided in this embodiment includes: As Figure 6 shown, the multi-modal answer generation device may vary greatly due to configuration or performance differences, and may include one or more processors 601 and a memory 602. One or more application programs or data may be stored in the memory 602. Among them, the memory 602 may be short-term storage or persistent storage. The application programs stored in the memory 602 may include one or more modules (not shown in the figure), and each module may include a series of computer-executable instructions in the multi-modal answer generation device. Further, the processor 601 may be set to communicate with the memory 602 to execute a series of computer-executable instructions in the memory 602 on the multi-modal answer generation device. The multi-modal answer generation device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input / output interfaces 605, one or more keyboards 606, etc.

[0090] In a specific embodiment, the multimodal answer generation device includes a memory and one or more programs. One or more of the programs are stored in the memory, and one or more of the programs may include one or more modules. Each module may include a series of computer-executable instructions in the multimodal answer generation device and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following: Input the question and prompt text into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer; Perform text retrieval in the text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain the initial text; Input the initial text into the large language model for text merging to obtain the answer text; Determine the positional association relationship between the multimedia data associated with the initial text and the answer text, and splice the multimedia data and the answer text according to the positional association relationship to obtain the multimodal answer data for the question.

[0091] An embodiment of the computer-readable storage medium provided in this specification is as follows: Corresponding to the multimodal answer generation method described above, based on the same technical concept, one or more embodiments of this specification also provide a computer-readable storage medium.

[0092] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, and the computer-executable instructions, when executed, implement the following process: Input the question and prompt text into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer; Perform text retrieval in the text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain the initial text; Input the initial text into the large language model for text merging to obtain the answer text; Determine the positional association relationship between the multimedia data associated with the initial text and the answer text, and splice the multimedia data and the answer text according to the positional association relationship to obtain the multimodal answer data for the question.

[0093] It should be noted that the embodiments of a computer-readable storage medium in this specification and the embodiments of a multi-modal answer generation method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method described above, and the repeated parts will not be elaborated.

[0094] An embodiment of a computer program product provided in this specification is as follows: Corresponding to the multi-modal answer generation method described above and based on the same technical concept, one or more embodiments of this specification also provide a computer program product.

[0095] A computer program product includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the following steps are implemented: Input the question and prompt text into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer; Perform text retrieval in a text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain initial text; Input the initial text into a large language model for text merging to obtain an answer text; Determine the positional association relationship between the multimedia data associated with the initial text and the answer text, and splice the multimedia data and the answer text according to the positional association relationship to obtain the multi-modal answer data of the question.

[0096] It should be noted that the embodiments of a computer program product in this specification and the embodiments of a multi-modal answer generation method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method described above, and the repeated parts will not be elaborated.

[0097] Each embodiment in this specification is described in a progressive manner. The same or similar parts between each embodiment can be referred to each other. The key point of each embodiment is to illustrate the differences from other embodiments. For example, the device embodiment, the equipment embodiment, and the computer-readable storage medium embodiment are all similar to the method embodiment, so the description is relatively simple. Please refer to the relevant description of the method embodiment for reading the relevant content in the device embodiment, the equipment embodiment, and the computer-readable storage medium embodiment.

[0098] The above description has been made of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multi-answer generation and parallel processing are also possible or may be advantageous.

[0099] In the 1930s, it was obvious to distinguish whether an improvement in technology was an improvement in hardware (e.g., improvement in circuit structures such as diodes, transistors, switches, etc.) or an improvement in software (improvement in method processes). However, with the development of technology, many improvements in method processes today can be regarded as direct improvements in hardware circuit structures. Almost all designers obtain the corresponding hardware circuit structures by programming the improved method processes into the hardware circuits. Therefore, it cannot be said that an improvement in a method process cannot be implemented with a hardware entity module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logical function is determined by the user programming the device. The designer can program by himself to "integrate" a digital system on a piece of PLD, without having to ask a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development writing, and the original code before compilation also has to be written in a specific programming language, which is called a Hardware Description Language (HDL), and there is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be clear that only by slightly logically programming the method process with the above-mentioned several hardware description languages and programming it into the integrated circuit, it is easy to obtain the hardware circuit implementing the logical method process.

[0100] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0101] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0102] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0103] Those skilled in the art should understand that one or more embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0104] This specification is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It should be understood that each flow and / or block in the flowchart and / or block diagram, and combinations of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0105] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are performed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0107] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0108] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.

[0109] A computer-readable medium includes permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0110] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising at least one..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0111] One or more embodiments of this specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data categories. One or more embodiments of this specification can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0112] The above are only embodiments of this document and are not used to limit this document. For those skilled in the art, this document can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A multimodal answer generation method, comprising: Inputting the question and the prompt text into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer; Performing text retrieval in a text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain an initial text; Inputting the initial text into a large language model for text merging to obtain an answer text; Determining the positional association relationship between the multimedia data associated with the initial text and the answer text, and splicing the multimedia data and the answer text according to the positional association relationship to obtain the multimodal answer data of the question.

2. The multimodal answer generation method according to claim 1, wherein the prompt text is used to instruct the large language model to perform question parsing and generate an answer according to the parsed question; Correspondingly, the question parsing includes: Splitting the question to obtain the at least one sub-question, and performing an association process on the historical question and the question to obtain the associated question.

3. The multimodal answer generation method according to claim 2, wherein the question parsing further includes: Correcting the question to obtain a corrected question, and / or extending the question to obtain an extended question.

4. The multimodal answer generation method according to claim 1, wherein the performing text retrieval in a text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain an initial text includes: Performing text retrieval in the text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain a plurality of candidate initial texts; Screening out the initial text whose similarity to the question meets the similarity condition from the plurality of candidate initial texts.

5. The multimodal answer generation method according to claim 4, wherein the screening out the initial text whose similarity to the question meets the similarity condition from the plurality of candidate initial texts includes: Inputting the question and each candidate initial text into a similarity calculation model for similarity calculation to obtain the semantic similarity between the question and each candidate initial text; Determining the candidate initial text with a semantic similarity greater than the similarity threshold as the initial text.

6. The multimodal answer generation method according to claim 1, wherein the determining the positional association relationship between the multimedia data associated with the initial text and the answer text includes: Inputting the multimedia data and each sub-text in the answer text into an index calculation model for association index calculation to obtain the association index between the multimedia data and each sub-text; Determining a target sub-text from the plurality of sub-texts according to the association index, and establishing the positional association relationship between the multimedia data and the target sub-text.

7. The multimodal answer generation method according to claim 1, wherein the text database contains a plurality of texts; each text in the plurality of texts and the associated multimedia data form multimodal data; The multi-modal data is generated in the following manner: The document multimedia data is locally stored to obtain a local storage identifier, and the document multimedia data in the document is replaced with the local storage identifier to obtain document data including the document text and the local storage identifier; The document data is partitioned to obtain a plurality of multi-modal data.

8. The multi-modal answer generation method according to claim 7, wherein the partitioning the document data to obtain a plurality of multi-modal data includes: Reading the multimedia text of the document multimedia data according to the local storage identifier; Performing feature extraction on the document text and the multimedia text in the document data to obtain a text feature matrix; Calculating a similarity matrix according to the text feature matrix and a preset constraint matrix, and partitioning the document data according to the similarity matrix to obtain a plurality of multi-modal data.

9. A multi-modal answer generation device, comprising: A question parsing module configured to input a question and prompt text into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer; A text retrieval module configured to perform text retrieval in a text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain initial text; A text merging module configured to input the initial text into a large language model for text merging to obtain an answer text; A multi-modal answer data splicing module configured to determine the positional association relationship between the multimedia data associated with the initial text and the answer text, and splice the multimedia data and the answer text according to the positional association relationship to obtain the multi-modal answer data of the question.

10. The multi-modal answer generation device according to claim 9, wherein the text retrieval module includes: A retrieval sub-module configured to perform text retrieval in the text database according to the question, each sub-question, the associated question, the sub-question answer, and the associated question answer to obtain a plurality of candidate initial texts; A filtering sub-module configured to screen out the initial text whose similarity to the question meets the similarity condition from the plurality of candidate initial texts.

11. The multi-modal answer generation device according to claim 10, wherein the filtering sub-module includes: A similarity calculation unit configured to input the question and each candidate initial text into a similarity calculation model for similarity calculation to obtain the semantic similarity between the question and each candidate initial text; An initial text determination unit configured to determine the candidate initial text with a semantic similarity greater than the similarity threshold as the initial text.

12. A multi-modal answer generation device, comprising: A processor; And a memory configured to store computer-executable instructions, the computer-executable instructions, when executed, cause the processor to: Input a question and prompt text into a large language model for question parsing and answer generation to obtain at least one sub-question, associated question, sub-question answer, and associated question answer; Perform a text search in the text database based on the problem, each sub-problem, the associated problem, the answers to the sub-problems, and the answers to the associated problem to obtain the initial text; Input the initial text into a large language model for text merging to obtain the answer text; Determine the positional association relationship between the multimedia data associated with the initial text and the answer text, and splice the multimedia data and the answer text according to the positional association relationship to obtain the multi-modal answer data for the problem.

13. A computer-readable storage medium for storing computer-executable instructions that, when executed, implement the steps of the method recited in claim 1.

Images