Question-answering processing method, and device, product and storage medium

Abstract

Provided in the embodiments of the present disclosure are a question-answering processing method, and a device, a product and a storage medium. In the question-answering processing method, after a query instruction is acquired, target knowledge information that matches the query instruction can be acquired from among a plurality of pieces of knowledge information in a knowledge base, and the query instruction and content-parsed text that corresponds to the target knowledge information are input into a large language model for question-answering processing, wherein the content-parsed text that corresponds to the target knowledge information is obtained by means of performing content parsing on a target document element that corresponds to the target knowledge information, and when the target document element comprises a document element of a non-text modality, content parsing is performed on the target document element before the target document element is input into the large language model, such that the document element of the non-text modality in the target document element can be understood by the large language model, so as to provide question-answering reference knowledge with a relatively high reliability for the large language model. Therefore, the accuracy of answering of the large language model for the query instruction can be improved.

Description

[0001] This disclosure claims priority to Chinese patent application number 202410232704.5, filed with the China Patent Office on February 29, 2024, entitled "Question and Answer Processing Method, Device, Product, and Storage Medium," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of computer technology, and more particularly to a question and answer processing method, device, product, and storage medium. Background: With the development of artificial intelligence, large language models (LLMs) have become increasingly widely used. In a typical application scenario, large language models can be integrated with traditional document retrieval methods to produce a retrieval question and answer system. Such question and answer systems typically segment documents into paragraphs in a specific manner, vectorize the paragraphs, and store them in a vector engine. Upon receiving a user's query, they match the most similar paragraphs to the user's question, and then use the LLM to answer the question based on the content of the recalled paragraphs. In some scenarios, documents may contain content and information in non-text modalities. However, large language models can only accept text sequence input and cannot process information in modalities other than text, resulting in an inability to accurately answer some user questions. Therefore, a new solution is needed. SUMMARY OF THE INVENTION Various aspects of the present disclosure provide a question-and-answer processing method, device, product, and storage medium to enhance the large language model's ability to process multimodal documents and improve the accuracy of responses to queries. An embodiment of the present disclosure provides a question-and-answer processing method, comprising: receiving a query; obtaining target knowledge information matching the query from multiple pieces of knowledge information in a knowledge base; obtaining a content-parsed text corresponding to the target knowledge information, the content-parsed text obtained by performing content parsing on target document elements corresponding to the target knowledge information, the target document elements including document elements in non-text modalities; and inputting the query and the content-parsed text corresponding to the target knowledge information into a large language model, so that the large language model outputs a response that matches the query based on the content-parsed text corresponding to the target knowledge information. Optionally, obtaining target knowledge information that matches the query instruction from multiple knowledge information in the knowledge base includes: obtaining text representation information of each of the multiple knowledge information in the knowledge base; calculating the matching degree between the query instruction and the text representation information of each of the multiple knowledge information to obtain the matching degree score between the query instruction and the multiple knowledge information; and determining the target knowledge information that matches the query instruction from the multiple knowledge information based on the matching degree scores between the query instruction and the multiple knowledge information.Optionally, calculating a match between the query instruction and the text representation information of each of the plurality of knowledge information to obtain a match score between the query instruction and each of the plurality of knowledge information includes: calculating a match between the text representation information of any of the plurality of knowledge information and the query instruction using multiple recall algorithms to obtain multiple match scores; determining weight coefficients corresponding to the multiple recall algorithms based on the modality of the knowledge information and the applicability of the multiple recall algorithms to knowledge information of different modalities; and performing a weighted calculation on the multiple match scores based on the weight coefficients corresponding to the multiple recall algorithms to obtain a match score between the query instruction and the knowledge information. Optionally, before obtaining the text representation information of each of the plurality of knowledge information in the knowledge base, the method further includes: obtaining multiple document elements from multiple document samples, the multiple document elements corresponding to multiple different modalities; adding the multiple document elements as knowledge information to the knowledge base; performing representation processing on the multiple document elements to obtain text representation information for each of the multiple document elements, and storing the correspondence between the multiple document elements and their respective text representation information. Optionally, characterization processing is performed on the multiple document elements to obtain text representation information of each of the multiple document elements, including: for any document sample among the multiple document samples, if the document sample includes a text paragraph, obtaining at least one information of the name of the document where the text paragraph is located, the paragraph title, the paragraph content, the document creation time, and the editing time as the text representation information of the text paragraph; if the document sample includes a table in text form, obtaining at least one information of the name of the document where the table is located, the table title, the key content in the table, the document creation time, and the editing time as the text representation information of the table; if the document sample includes a picture, obtaining at least one information of the name of the document where the picture is located, the picture title, the question identified from the picture, the description information of the picture, the document creation time, and the editing time as the text representation information of the picture. Optionally, it also includes: parsing the multiple document elements according to the modalities corresponding to each of the multiple document elements to obtain content parsed texts corresponding to each of the multiple document elements; obtaining multiple question-answer pair samples associated with the multiple document samples; any question-answer pair sample includes a query instruction sample and a corresponding reply result sample; obtaining multiple sample pairs according to the content parsed texts corresponding to each of the multiple document elements and the multiple question-answer pair samples; and training the large language model based on the multiple sample pairs.Optionally, parsing the multiple document elements according to the modalities corresponding to the document elements in the multiple document samples to obtain content parsed text corresponding to the multiple document elements includes at least one of the following: for any document sample among the multiple document samples, if the document sample includes a text paragraph, performing text segmentation on the text paragraph to obtain multiple text blocks as the content parsed text corresponding to the text paragraph; if the document sample includes a table in text form, identifying the text content and the logical relationship between cells in the text-form table, and encoding according to the identified text content and logical relationship to obtain encoding information as the content parsed text corresponding to the text-form table; if the document sample includes a picture, and the picture includes a table in picture form, identifying the text content and the positions of cell vertices in the picture-form table, determining the logical relationship between cells in the picture-form table according to the identified cell vertices, and encoding according to the identified text content and logical relationship to obtain encoding information as the content parsed text corresponding to the picture-form table; if the document sample includes a picture, and the picture does not include a table, Extracting text from the image or generating text description information based on the image as the content parsed text corresponding to the image. Optionally, obtaining multiple question-answer pair samples associated with the multiple document samples includes: inputting the multiple document samples and preset prompt words into a question-answering model, so that the question-answering model, under the guidance of the prompt words, generates the multiple question-answer pair samples based on the text content in the multiple document samples. Optionally, obtaining multiple sample pairs based on the content parsed text corresponding to each of the multiple document elements and the multiple question-answer pair samples includes: using a set recall algorithm for any query instruction sample in the multiple question-answer pair samples to determine a target document sample from the multiple document samples that matches the query instruction sample; determining a document element that matches the query instruction sample based on the document elements contained in the target document sample; using the document element that matches the query instruction sample as the knowledge information corresponding to the query instruction; and obtaining a sample pair corresponding to the query instruction sample based on the content parsed text of the question-answer pair sample to which the query instruction belongs and the knowledge information corresponding to the query instruction. Optionally, the document element adapted for the query instruction sample includes document elements of at least two different modalities. Embodiments of the present disclosure also provide an electronic device, comprising: a memory and a processor; the memory being configured to store one or more computer instructions; and the processor being configured to execute the one or more computer instructions to perform the steps of the method provided in embodiments of the present disclosure.The present disclosure also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program can implement the steps of the method provided in the present disclosure. The present disclosure also provides a computer program product, including a computer program / instructions. When executed by a processor, the computer program / instructions can implement the steps of the method provided in the present disclosure. In the question-answering processing method provided in the present disclosure, after receiving a query instruction, target knowledge information matching the query instruction can be obtained from multiple pieces of knowledge information in a knowledge base. The query instruction and the content-parsed text corresponding to the target knowledge information are then input into a large language model for question-answering processing. The content-parsed text corresponding to the target knowledge information is obtained by performing content parsing on the target document elements corresponding to the target knowledge information. When the target document elements include document elements in non-text modalities, content parsing of the target document elements before inputting them into the large language model allows the large language model to understand the non-textual document elements in the target document elements, thereby providing the large language model with highly reliable reference knowledge for question-answering. This can improve the knowledge comprehension and question-answering capabilities of the large language model, and enhance the accuracy of the large language model's responses to queries. BRIEF DESCRIPTION OF THE DRAWINGS The drawings described herein are intended to provide a further understanding of the present disclosure and constitute a part of this disclosure. The exemplary embodiments of this disclosure and their descriptions are intended to explain the disclosure and do not constitute undue limitations of the disclosure. In the drawings: Figure 1 is a flowchart of a question-answering processing method provided by an exemplary embodiment of the present disclosure; Figure 2 is a flowchart of processing document elements in image modality provided by an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram of a question-answering processing framework and question-answering processing flow provided by an exemplary embodiment of the present disclosure; and Figure 4 is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION To further clarify the objectives, technical solutions, and advantages of this disclosure, the technical solutions of this disclosure will be described clearly and completely below in conjunction with the specific embodiments of this disclosure and the corresponding drawings. Obviously, the described embodiments represent only a portion of the embodiments of this disclosure, and not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. The terminology used in the embodiments of the present disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. As used in the embodiments of the present disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms. Unless the context clearly indicates otherwise, "a plurality" generally includes at least two, but does not exclude the inclusion of at least one.It should be understood that the term "and / or" as used herein merely describes an association relationship between associated objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship. It should also be noted that the terms "include," "comprise," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a set of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or system. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the product or system comprising the element. In some scenarios, a document may contain content and information in multiple modalities, such as text paragraphs, tables in the form of text, tables in the form of images, or other non-tabular images. However, large language models can only accept text sequences as input and cannot process information in other modalities besides text, which in turn leads to an inability to accurately answer some user questions. In some question-answering systems, if a document contains images, the images are often discarded or only the text information in the images is extracted based on OCR (Optical Character Recognition). If a document contains tables, the tables are often forcibly converted into text paragraphs. This approach can lead to significant information loss, misalignment of text logical relationships, and other consequences, ultimately significantly reducing the scope of questions and answers or decreasing the accuracy of question-answering. To address the above technical issues, some embodiments of the present disclosure provide a solution. The technical solutions provided by various embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. Figure 1 is a flowchart of a question-answering processing method provided by an exemplary embodiment of the present disclosure. As shown in Figure 1, the method primarily includes the following steps: Step 101: Receive a query instruction. Step 102: Obtain target knowledge information matching the query instruction from multiple pieces of knowledge information in a knowledge base. Step 103: Obtain the content parsed text corresponding to the target knowledge information. The content parsed text is obtained by performing content parsing on target document elements corresponding to the target knowledge information, including document elements in non-text modalities. Step 104: Input the query instruction and the content parsed text corresponding to the target knowledge information into a large language model, so that the large language model outputs a response result adapted to the query instruction based on the content parsed text corresponding to the target knowledge information.The query instruction can be issued by a user or by an application according to predefined processing logic, which is not a limitation in this embodiment. The query instruction can be a selective query instruction, a summary query instruction, or a rejection query instruction. A knowledge base is a system or tool for centrally storing, organizing, and managing knowledge. It is typically used to collect, organize, and provide knowledge information about specific topics, fields, or industries. The knowledge information in a knowledge base can include various types of knowledge content, such as documents, images, videos, and audio data, allowing users or other applications to easily retrieve the required knowledge through retrieval. Any document can contain document elements of various modalities. A document element refers to the various components that make up a document. In this embodiment, document element modalities can include text paragraphs, text-based tables, image-based tables, and images. The parsed text of a document element's content refers to the textual information obtained by parsing the content contained in the document element. This textual information is used to describe the content of document elements and the logical relationships between them in text form, enabling large language models to understand non-textual document elements. Documents in the knowledge base can be text documents, Word documents, PDF documents, Excel documents, and so on. Tables in these types of documents can be textual or image-based. A textual table is one created directly in document editing software, such as a table created in a Word document using the table function in Word software. A textual table is a form of semi-structured data that contains a large amount of numerical values ​​and text. Cells have certain logical relationships, which can be simple or complex. The logical relationships between cells can include correspondences between fields and values ​​in different cells, or include-and-be-included relationships between fields in different cells. These include-and-be-included relationships can be represented by merging cells. An image-based table presents a table as an image. A table in image format can be created in spreadsheet software (such as Excel) or graphic design software, then captured or exported as an image format before being inserted into a document. In a knowledge base, individual document elements can also serve as knowledge information for retrieval and matching. In this embodiment, after receiving a query instruction, the knowledge base can be searched based on the query instruction to obtain target knowledge information matching the query instruction from multiple pieces of knowledge information in the knowledge base.The retrieved target knowledge information may include document elements in one or more modalities. The query instruction is typically in textual modality. In some optional embodiments, the knowledge information in the knowledge base may be converted into information in textual modality. When searching the knowledge base based on the query instruction, the target knowledge information may be retrieved by calculating the matching degree between the query instruction in textual modality and the knowledge information in textual modality. An exemplary explanation is provided below. Optionally, textual representation information corresponding to each of multiple pieces of knowledge information in the knowledge base may be obtained, and a matching degree calculation may be performed between the query instruction and the textual representation information of each piece of knowledge information to obtain a matching score between the query instruction and each piece of knowledge information. The textual representation information is used to describe the knowledge information in textual form. When constructing the knowledge base, a predetermined information representation method may be used to obtain textual representation information for each piece of knowledge information in the knowledge base. Each piece of knowledge information and its corresponding textual representation information may be stored in correspondence for subsequent access. In this embodiment, when the modalities of the knowledge information in the knowledge base are diverse, text representation information corresponding to the knowledge information can be obtained. Therefore, when searching the knowledge base based on a query instruction, a matching degree calculation can be performed between the query instruction in the text modality and the text representation information in the text modality. This allows for an accurate understanding of the meaning of the query instruction and the knowledge information in a unified modality, thereby improving the accuracy of the matching degree calculation results. The matching degree calculation operation for the query instruction in the text modality and the text representation information in the text modality can be implemented using a recall algorithm. For example, this recall algorithm can be a vector recall algorithm, a BM25 recall algorithm, or a TF-DF (Term Frequency-Inverse Document Frequency) recall algorithm. Vector recall is a recall method based on vector similarity calculation. Its core concept is to represent the features to be calculated using vectors and then calculate feature similarity based on the vectors. The vector recall process can be divided into two steps: feature vectorization and vector similarity calculation. First, the features to be calculated can be converted into vector form using a deep learning model (such as word embedding or the BERT model). Then, the similarity between these vectors is calculated in the vector space. In this embodiment, after obtaining the textual representation information of the knowledge information in the knowledge base, the encoding model is used to vectorize the textual representation information of the knowledge information, and the resulting vectors are stored in the vector engine.When calculating the match between a query and the textual representation of any piece of knowledge, the query vector and the textual representation vector are obtained. The similarity between the query vector and the textual representation vector is then calculated through vector computation. BM25 is a statistically based ranking function algorithm that achieves recall by weighting each word in a document and then calculating the similarity between the query and the document. The core concept of the BM25 algorithm is to determine the weight of each word based on term frequency (TF) and inverse document frequency (IDF). Term frequency indicates the number of times a word appears in a document, while inverse document frequency indicates the rarity of a word in the entire document collection. By combining these two factors, BM25 can assess the importance of a word in a specific document and calculate the similarity between the query and the document accordingly. TF-IDF is an algorithm that calculates term frequency and inverse document frequency and multiplies them together to obtain a word's importance score within the entire document collection. During the recall phase, TF-IDF evaluates the similarity between the query (i.e., the words in the query) and the words in the document to achieve recall. Different recall algorithms can be applied to recall knowledge information of different modalities. Therefore, in some optional embodiments, when searching a knowledge base based on a query, a multi-way recall approach can be employed to retrieve optimal knowledge information from the knowledge base. The following description uses any one of these multiple pieces of knowledge information as an example. Optionally, when calculating the matching degree between the query and the textual representation of the knowledge information, multiple recall algorithms can be employed to calculate the matching degree between the textual representation of the knowledge information and the query, resulting in multiple matching scores. When multiple recall algorithms are used to calculate the matching scores between the query and the same piece of knowledge information, the reliability of the matching scores corresponding to different recall algorithms varies, indicating that different recall algorithms are applicable to knowledge information of different modalities. Therefore, when determining the modality of the knowledge information, the weight coefficients corresponding to the various recall algorithms can be determined based on their applicability to knowledge information of different modalities. For example, if the knowledge information is a text paragraph, the weights for the various recall algorithms are w1, w2, and w3, respectively; if the knowledge information is a table in the form of text, the weights for the various recall algorithms are w4, w5, and w6, respectively; and if the knowledge information is a table in the form of an image, the weights for the various recall algorithms are w7, w8, and w9, respectively.Based on the weight coefficients corresponding to the multiple recall algorithms, a weighted calculation can be performed on the multiple matching scores to obtain a matching score between the query and the knowledge information. After obtaining the matching scores between the query and the multiple pieces of knowledge information in the knowledge base based on the above embodiment, the target knowledge information that matches the query can be determined from the multiple pieces of knowledge information based on the matching scores between the query and the multiple pieces of knowledge information. The target knowledge information can be the piece of knowledge information with the highest matching score, or multiple pieces of knowledge information with high matching scores, although this embodiment is not limiting. When the target knowledge information includes multiple pieces of knowledge information, the modalities of the multiple pieces of knowledge information can be the same or different. After determining the target knowledge information based on the above embodiment, the content parsed text corresponding to the target knowledge information can be obtained, and the query and the content parsed text corresponding to the target knowledge information can be input into a large language model. In this embodiment, a large language model refers to a large language model with a parameter count greater than a set parameter count threshold. The content parsed text of the target knowledge information is obtained by performing content parsing on the target document element corresponding to the target knowledge information, which includes document elements in non-text modalities. In this embodiment, the training corpus used in the pre-training process of the large language model contains a large amount of text and encoded information of textual morphology, enabling the large language model to understand the content parsed text of document elements. Based on this, after a query instruction and the content parsed text corresponding to the target knowledge information are input into the large language model, the large language model can output a response tailored to the query instruction based on the content parsed text corresponding to the target knowledge information, using learned parameters. In this embodiment, after receiving a query instruction, target knowledge information matching the query instruction is retrieved from multiple pieces of knowledge information in the knowledge base. The query instruction and the content parsed text corresponding to the target knowledge information are then input into the large language model for question-answering processing. The content parsed text corresponding to the target knowledge information is obtained by performing content parsing on the target document element corresponding to the target knowledge information. If the target document element includes document elements in non-text modalities, content parsing of the target document element before inputting it into the large language model ensures that the non-textual document elements in the target document element can be understood by the large language model, providing the large language model with highly reliable reference knowledge for question-answering. This can improve the knowledge comprehension and question-answering capabilities of the large language model, and enhance the accuracy of its responses to queries. In some optional embodiments, a knowledge base can be constructed or updated based on multiple document samples. In this implementation, multiple document elements can be obtained from the multiple document samples, where the multiple document elements correspond to multiple different modalities.The multiple document samples may be various documents from multiple different application fields, covering various types of knowledge in various application scenarios of the large language model. After obtaining the multiple document elements contained in the multiple document samples, the multiple document elements may be added to the knowledge base as knowledge information. Optionally, the multiple document elements may be characterized to obtain textual representation information for each of the multiple document elements, and the corresponding relationship between the multiple document elements and their respective textual representation information may be stored. The textual representation information of document elements in different modalities may be obtained in different ways. The following will use any of the multiple document samples as an example to illustrate the characterization processing of document elements. If the document sample includes a text paragraph, at least one of the following information is obtained: the name of the document containing the text paragraph, the paragraph title, the paragraph content, the document creation time, and the editing time; the textual representation information of the text paragraph. If the document sample includes a table in text form, at least one of the following information is obtained: the name of the document containing the table, the table title, key content within the table, the document creation time, and the editing time; the textual representation information of the table. The key content of the table may be a portion of the fields within the table. If the document sample includes an image, at least one of the following information is obtained: the document name containing the image, the image title, the question identified from the image, the image description, the document creation time, and the editing time, as the text representation information of the image. The correspondence between the multiple document elements and their respective text representation information may be stored in a knowledge base or in another mapping table, which is not a limitation in this embodiment. When searching the knowledge base based on a query instruction, the text representation information of each document element in the knowledge base may be read based on the above correspondence, and the degree of match between the query instruction and the text representation information of the document element may be calculated. It is worth noting that in some optional embodiments, the general large language model may be further enhanced and trained to make it more suitable for search-based question-answering services that integrate multimodal information. The following provides an exemplary description of the enhanced training process for the large language model. In this embodiment, a pre-trained general large language model may be used as a base model, and the base model is further pre-trained using document samples from multiple different domains to train the base model's ability to understand the content of document elements and parse text. When training a large language model, multiple document elements can be obtained from multiple document samples, where the multiple document elements correspond to multiple different modalities. The multiple document samples can be documents from multiple different application domains, thereby training the large language model's ability to understand document elements in a general domain.After obtaining multiple document elements, the multiple document elements can be parsed according to the modalities corresponding to each of the multiple document elements to obtain the content parsed text corresponding to each of the multiple document elements. The following will use any document sample as an example to illustrate optional implementations of parsing document elements in the document sample. Optionally, if the document sample includes a text paragraph, the text paragraph is segmented to obtain multiple text blocks, which serve as the content parsed text corresponding to the text paragraph. Optionally, if the document sample includes a table in text form, the text content and logical relationships between cells in the text form table are identified, and encoding is performed based on the identified text content and logical relationships to obtain encoding information, which serves as the content parsed text corresponding to the text form table. In some optional embodiments, the text form table can be encoded into at least one of HTML encoding, LaTeX encoding, and Markdown encoding based on the identified text content and logical relationships. The pre-training corpus for a large language model typically contains a large amount of HTML and LaTeX code, so the large language model has a good understanding of this type of encoding. This approach can fully utilize the large language model's ability to understand such encodings and convert tables into encoding, enabling the large language model to understand tabular data. Optionally, if the document sample includes an image, and the image includes a table in the form of an image, the text content and the positions of cell vertices in the image-based table are identified. The logical relationships between cells in the image-based table are determined based on the identified cell vertices. Encoding is performed based on the identified text content and logical relationships to obtain encoding information, which is used as the content parsed text corresponding to the image-based table. In some embodiments, as shown in FIG2 , when the image includes a table in the form of an image, the image-based table can be encoded based on the image model. In this embodiment, samples can be constructed based on image-based tables and the corresponding true encoding values ​​of the tables. The constructed samples can then be used to perform supervised training on the image model. This allows the image model to learn to identify text content and logical relationships within the input image-based tables and to encode the identified text content and logical relationships. Alternatively, if the document sample includes an image but does not include a table, text can be extracted from the image or text description information can be generated based on the image to serve as the content parsed text corresponding to the image. The image typically refers to an image that does not include a table. Extracting text from the image can be accomplished using OCR technology, while generating text description information from the image can be implemented using an image-to-text model.As shown in Figure 2, when the image does not contain an image, text within the image can be extracted using OCR technology, or a text-to-image model can be used to generate text description information corresponding to the image. Of course, in some embodiments, the image can also be displayed to the user, and a text description input by the user based on the image can be obtained. Optionally, multiple question-answer pair samples associated with the multiple document samples can be obtained, with each question-answer pair sample comprising a query instruction sample and a corresponding response result sample. The query instruction sample simulates a question instruction issued by a user in an actual application scenario, and the response result sample simulates the response result given by the large language model to the query instruction. The question-answer pair sample can be constructed by the user based on the content of the document sample, or automatically generated based on the question-answer model. In some optional embodiments, the multiple document samples and preset prompt words can be input into the question-answer model, so that the question-answer model, under the guidance of the prompt words, generates multiple query instruction samples and corresponding response result samples based on the text content of the multiple document samples. Optionally, after obtaining multiple query instruction samples and corresponding answer result samples output by the question-answering model, these multiple query instruction samples and corresponding answer result samples can be manually reviewed to improve the reliability of the question-answer pair samples, thereby enhancing the enhanced training effect of the large language model. Based on the above-mentioned embodiment, after obtaining the content parsed text corresponding to multiple document elements and multiple question-answer pair samples, multiple sample pairs can be obtained based on the content parsed text corresponding to each of the multiple document elements and the multiple question-answer pair samples. A sample pair can include a question-answer pair sample and the content parsed text corresponding to at least one document element. The content parsed text corresponding to the at least one document element is the knowledge information required to generate the question-answer pair sample. That is, based on this sample pair, the large language model can learn the ability to generate answer result samples based on the content parsed text corresponding to the at least one document element and the query instruction samples in the question-answer pair sample. In some optional embodiments, sample pairs can be manually constructed based on the matching relationship between the question-answer pair sample and the document elements. In other optional embodiments, sample pairs can be constructed by a machine model based on the matching relationship between the question-answer pair sample and the document elements. The following description uses any one of the multiple question-answer pair samples as an example. Optionally, for a query instruction sample in the question-answer pair sample, the machine model may employ a predefined recall algorithm to determine a target document sample from the multiple document samples that matches the query instruction sample. This recall algorithm may be the vector recall algorithm, BM25 recall algorithm, TF-IDF recall algorithm, or the like described in the previous embodiments.After determining the target document sample, the machine model can determine the document elements that match the query sample based on the document elements contained in the target document sample. In this step, the machine model can calculate the matching score between each document element in the target document and the query sample, and based on the matching scores, determine the document elements that match the query sample from the target document sample. The document elements that match the query sample can then be used as the knowledge information corresponding to the query. The machine model can then parse the text based on the sample question-answer pairs associated with the query and the content of the knowledge information corresponding to the query to obtain sample pairs corresponding to the query sample. In some optional embodiments, after the machine model outputs the document elements that match the query sample, they can be manually reviewed to improve the accuracy of the sample pairs. In some optional embodiments, the document elements that match the query sample can include document elements of a single modality to simulate a simple question-and-answer scenario. For example, the document element adapted to the query instruction sample may be a text paragraph, a table in text form, a table in image form, or an image. In other optional embodiments, the document element adapted to the query instruction sample may include document elements of at least two different modalities to simulate complex question-and-answer scenarios. For example, the document element adapted to the query instruction sample may be a combination of a text paragraph and an image, or a combination of a text paragraph and a table in image form, or a combination of a text paragraph, a table in text form, and an image, to name a few. After obtaining multiple sample pairs based on the above embodiment, the large language model can be trained based on these multiple sample pairs. It is worth noting that to simulate different question-and-answer scenarios, different types of tasks can be constructed for the large language model. Different types of tasks can be constructed by adding prompt words corresponding to the task type to the query instruction sample. Task types may include: selective question-and-answer tasks, summarizing question-and-answer tasks, and rejection question-and-answer tasks. Selective question-and-answer tasks are used to select a reference from multiple input references as a response. The summary-based question answering task is used to integrate and summarize multiple reference information and use the summary as the answer. The rejection-based question answering task is used to refuse to answer or use a fallback response when the reference information is determined to be incorrect. This will not be described in detail here. After the base model converges using the above training method, the converged model can be fine-tuned using document samples from a specific domain to ensure that the fine-tuned model has better performance in question answering tasks in that specific domain.It should be understood that during the fine-tuning phase, samples matching specific tasks in a vertical domain can be constructed using the aforementioned implementations based on documents in the vertical domain, and the model can be trained using the aforementioned training methods to achieve fine-tuning. After the fine-tuned model is put into use, further reinforcement learning can be performed on the fine-tuned model based on user feedback on the model's output responses, gradually improving model performance. The following further illustrates the question-answering processing method provided by the present disclosure in conjunction with the architecture flow chart shown in FIG3 . As shown in FIG3 , the question-answering processing framework primarily includes a knowledge acquisition module, a document parsing module, an information representation module, a multi-way recall module, a large-model question-answering module, and a reinforcement training module. The knowledge acquisition module is used to acquire multimodal documents. Multimodal documents may include, but are not limited to, text files, Word documents, PDF documents, Excel files, images, web pages, and the like. Document content can be categorized into different content formats, such as text paragraphs, text-based tables, image-based tables, and other images. Any document can contain one or more of these different content formats. In the document parsing module, various forms of content can be identified and parsed using different processing steps. Different form recognition methods can be used depending on the document format. For example, for Word documents, Word processing toolkits (such as those based on Python or Java) can be used to extract text paragraphs, tables, images, etc. from Word documents. The following will provide an exemplary explanation of the parsing steps shown in Figure 3. As shown in Figure 3, the parsing operation for a multimodal document may include the following steps:

[0002] 1. For text paragraphs, split the document into paragraphs or fixed lengths (such as 512 characters) to obtain text blocks.

[0003] 2. For text-based tables, extract the content in the table and the logical relationship between cells, and use one or more encoding methods to encode the content in the table and the logical relationship between cells to obtain the table code.

[0004] 3. For tables in the form of images, an image model can be used to extract the table's content and the logical relationships between cells. These content and the logical relationships between cells can then be encoded using one or more encoding methods to generate a table code. 4. For images that do not contain tables, an image model can be used to extract textual information from the image, or an image-text generation model can be used to generate textual descriptions corresponding to the image. Alternatively, manual editing can be used to generate textual descriptions corresponding to the image (not shown). In the information representation module, different representation methods can be used for different content types. As shown in Figure 3, the document information representation operation may include the following steps:

[0005] 5. For a text paragraph, the document name, paragraph title, paragraph content, document creation time, and editing time of the paragraph are used as the text representation information of the text paragraph.

[0006] 6 and 7. For text-based tables and image-based tables, the document name, table title, key content in the table (such as field names), document creation time, and editing time can be used as the text representation information of the table.

[0007] 8. For other images containing tables, the text representation information for the image can include the document name, image title, text extracted from the image, text description information generated by the image-to-text model, manually edited description, document creation time, and editing time. In the multi-channel recall process, based on the received user question, the text representation information matching the user question can be obtained from the information representations obtained in the previous steps. As shown in Figure 3, the multi-channel recall operation may include the following steps:

[0008] 9. Provide retrieval and matching services using multi-way recall and fusion scoring. Multi-way recall methods may include, but are not limited to, vector recall, BM25 recall, and TF-IDF. After obtaining text representation information corresponding to document elements of different modalities based on the information representation component, this text representation information can be vectorized using an encoding model, and the resulting vectors can be stored in the vector engine. After receiving a user question, various recall methods can be used based on the user's question to retrieve one or more document elements that are most similar to the user's question as the recall result.

[0009] 10. Fuse and score the multi-way recall results, then sort them based on the scoring results. Based on the sorting results, select the recall result with the highest score, or several results with high scores, from the recall results. This fusion scoring method can be based on a weighted formula or a ranking model (such as a BERT-based twin-tower model). In the large-model question-answering module, as shown in Figure 3, step 11 can be performed to input the user question and the highest-scoring recall result, or several results with high scores, into the large language model for question-answering. In the enhanced training module, specific samples and tasks can be constructed to perform continued pre-training, fine-tuning, and reinforcement learning based on human feedback on the general large language model to further improve its question-answering performance. As shown in Figure 3, the enhanced training module performs enhanced training on the large language model by performing the following steps: Step 12: Continue pre-training on the general large language model based on multimodal sample data. For example, table encoding information can be used to perform further pre-training on a general large language model to further improve its understanding of table encodings. In this step, sample pairs and question-answering tasks can be constructed based on the application scenario. Training samples that meet the application scenario can be retrieval-based question-answering samples that integrate multimodal information. The modalities of the knowledge information in any sample pair can include the following combinations: a simple paragraph, image, or table; a paragraph + table; a paragraph + image; an image + table; or a combination of a paragraph + image + table, not listed here. Question-answering tasks can include selective question-answering tasks, summarizing question-answering tasks, or rejection question-answering tasks. When constructing sample pairs, sample pairs can be manually curated. Alternatively, a machine-based approach with manual support can be used to curate sample pairs. When constructing sample pairs using a machine, a question-answering model can be used to generate a batch of question-answer pairs based on the content of the document samples. The questions in the question-answer pairs can then be used to retrieve documents from the document library that are most similar to the questions. The most similar document can be used as the knowledge information corresponding to the question and answer. Based on the question and the recalled knowledge information in the question-answer pair, sample pairs can be constructed. After the recalled documents are retrieved from the document library, human assistance can be used to verify the match between the question and the recalled documents to improve the accuracy of the generated sample pairs. In step 13, based on the question-answer pairs constructed in step 11, the large language model is fine-tuned and improved. In step 14, reinforcement learning based on human feedback is performed on the fine-tuned large language model. In this step, a scoring model can be trained based on human feedback data, such as reflux data from real-world question-answering scenarios and user rating data.After the large language model outputs each response, the scoring model scores it, and reinforcement learning is used to further fine-tune the large model based on the scoring results. In this implementation, encoding tables in text and image formats fully preserves the table's content and the logical relationships between cells, significantly improving the large language model's understanding of this information. For images that don't contain tables, text recognition based on OCR technology or the image-to-text model generates text descriptions of the image. This fully preserves the information carried by the image, allowing the large language model to understand and fully utilize diverse information to generate responses. This implementation allows a larger number of documents with complex modalities to be applied to search-based question answering tasks, thereby expanding the scope of knowledge and improving the accuracy of search-based question answering. It should be noted that the execution of each step of the method provided in the above embodiment can be performed by the same device, or the method can be performed by different devices. For example, steps 101 to 104 may be performed by device A; another example, steps 101 and 102 may be performed by device A, and step 103 may be performed by device B; and so on. Furthermore, some of the processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be understood that these operations may be performed in a different order or in parallel. Operation numbers, such as 101 and 102, are merely used to distinguish between different operations and do not represent any specific execution order. Furthermore, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel. It should be noted that terms such as "first" and "second" are used herein to distinguish between different messages, devices, modules, and the like, and do not imply a sequential order or limit "first" and "second" to different types. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or reject. Figure 4 shows a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present disclosure. As shown in Figure 4, the electronic device includes: memory 401, processor 402, and communication component 403.Memory 401 is configured to store computer programs and may be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. Processor 402, coupled to memory 401, is configured to execute the computer program in memory 401 to: receive a query instruction; obtain target knowledge information matching the query instruction from multiple pieces of knowledge information in a knowledge base; obtain content-parsed text corresponding to the target knowledge information, the content-parsed text being obtained by performing content parsing on target document elements corresponding to the target knowledge information, the target document elements including document elements in non-textual modalities; and input the query instruction and the content-parsed text corresponding to the target knowledge information into a large language model, so that the large language model outputs a response result adapted to the query instruction based on the content-parsed text corresponding to the target knowledge information. Optionally, when the processor 402 obtains target knowledge information that matches the query instruction from multiple knowledge information in the knowledge base, it is specifically used to: obtain text representation information of each of the multiple knowledge information in the knowledge base; calculate the matching degree between the query instruction and the text representation information of each of the multiple knowledge information to obtain the matching degree score between the query instruction and the multiple knowledge information; and determine the target knowledge information that matches the query instruction from the multiple knowledge information based on the matching degree scores between the query instruction and the multiple knowledge information. Optionally, when the processor 402 performs a matching calculation between the query instruction and the text representation information of each of the multiple knowledge information to obtain a matching score between the query instruction and the multiple knowledge information, it is specifically used to: for any knowledge information among the multiple knowledge information, use multiple recall algorithms to perform a matching calculation between the text representation information of the knowledge information and the query instruction to obtain multiple matching scores; determine the weight coefficients corresponding to the multiple recall algorithms according to the modality of the knowledge information and the applicability relationship between the multiple recall algorithms and knowledge information of different modalities; perform a weighted calculation on the multiple matching scores according to the weight coefficients corresponding to the multiple recall algorithms to obtain a matching score between the query instruction and the knowledge information. Optionally, before obtaining the text representation information of each of the multiple knowledge information in the knowledge base, the processor 402 is further configured to: obtain the multiple document elements from multiple document samples, where the multiple document elements correspond to multiple different modalities; add the multiple document elements as knowledge information to the knowledge base; perform representation processing on the multiple document elements to obtain the text representation information of each of the multiple document elements, and save the correspondence between the multiple document elements and their respective text representation information.Optionally, when the processor 402 performs characterization processing on the multiple document elements and obtains text representation information of each of the multiple document elements, it is specifically used to: for any document sample among the multiple document samples, if the document sample includes a text paragraph, obtain at least one information of the name of the document where the text paragraph is located, the paragraph title, the paragraph content, the document creation time, and the editing time as the text representation information of the text paragraph; if the document sample includes a table in text form, obtain at least one information of the name of the document where the table is located, the table title, the key content in the table, the document creation time, and the editing time as the text representation information of the table; if the document sample includes a picture, obtain at least one information of the name of the document where the picture is located, the picture title, the question identified from the picture, the description information of the picture, the document creation time, and the editing time as the text representation information of the picture. Optionally, the processor 402 is further used to: parse the multiple document elements according to the modalities corresponding to each of the multiple document elements to obtain content parsed texts corresponding to each of the multiple document elements; obtain multiple question-answer pair samples associated with the multiple document samples; any question-answer pair sample includes a query instruction sample and a corresponding reply result sample; obtain multiple sample pairs according to the content parsed texts corresponding to each of the multiple document elements and the multiple question-answer pair samples; and train the large language model based on the multiple sample pairs.Optionally, when the processor 402 parses the multiple document elements according to the modalities corresponding to the document elements in the multiple document samples and obtains the content parsed texts corresponding to the multiple document elements, it is specifically configured to perform at least one of the following operations: for any document sample among the multiple document samples, if the document sample includes a text paragraph, performing text segmentation on the text paragraph to obtain multiple text blocks as the content parsed text corresponding to the text paragraph; if the document sample includes a table in text form, identifying the text content and the logical relationship between cells in the text-form table, and encoding according to the identified text content and logical relationship to obtain encoding information as the content parsed text corresponding to the text-form table; if the document sample includes a picture, and the picture includes a table in picture form, identifying the text content and the positions of the cell vertices in the picture-form table, determining the logical relationship between the cells in the picture-form table according to the identified cell vertices, and encoding according to the identified text content and logical relationship to obtain encoding information as the content parsed text corresponding to the picture-form table; if the document sample includes a picture, If the image does not include a table, text is extracted from the image or text description information is generated based on the image as the content parsed text corresponding to the image. Optionally, when obtaining multiple question-answer pair samples associated with the multiple document samples, processor 402 is specifically configured to: input the multiple document samples and preset prompt words into a question-answering model, so that the question-answering model, under the guidance of the prompt words, generates the multiple question-answer pair samples based on the text content in the multiple document samples. Optionally, when parsing the text corresponding to the content of each of the multiple document elements and the multiple question-answer pair samples to obtain multiple sample pairs, the processor 402 is specifically configured to: for any query instruction sample in the multiple question-answer pair samples, employ a set recall algorithm to determine a target document sample from the multiple document samples that matches the query instruction sample; determine a document element that matches the query instruction sample based on the document elements contained in the target document sample; use the document element that matches the query instruction sample as the knowledge information corresponding to the query instruction, and parse the text based on the content of the question-answer pair sample to which the query instruction belongs and the knowledge information corresponding to the query instruction to obtain a sample pair corresponding to the query instruction sample. Optionally, the document element that matches the query instruction sample includes document elements of at least two different modalities. Furthermore, as shown in FIG. 4 , the electronic device also includes other components, such as a power supply component 404, a display component 405, and an audio component 406.FIG4 schematically illustrates only some components, and does not mean that the electronic device includes only the components shown in FIG4 . Memory 401 may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Communication component 403 is configured to facilitate wired or wireless communication between the device in which the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as Wi-Fi (wireless network communication technology), 2G (such as Global System for Mobile Communications (GSM)), 3G (such as Wideband Code Division Multiple Access (WCDMA), 4G (such as Long Term Evolution (LTE)), 4G+ (such as upgraded version of Long Term Evolution (LTE-Advanced, LTE-A)), or 5G (fifth generation mobile communication technology (5th Generation Mobility Communications Technology)), or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel.In an exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IRDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies. The power supply component 404 is configured to provide power to various components of the device in which the power supply component resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device in which the power supply component resides. The display component 405 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or sliding action, but also detect the duration and pressure associated with the touch or sliding action. The audio component 406 can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC). When the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals. In this embodiment, after obtaining a query instruction, target knowledge information matching the query instruction can be obtained from multiple pieces of knowledge information in the knowledge base, and the query instruction and the content parsed text corresponding to the target knowledge information can be input into the large language model for question-and-answer processing. Among them, the content parsed text corresponding to the target knowledge information is obtained by performing content parsing on the target document elements corresponding to the target knowledge information. When the target document elements include document elements in non-text modalities, the target document elements are content parsed before being input into the large language model, so that the document elements in non-text modalities in the target document elements can be understood by the large language model, thereby providing the large language model with question-and-answer reference knowledge with higher reliability.This can improve the knowledge comprehension and question-answering capabilities of a large language model, and enhance the accuracy of the large language model's responses to queries. Accordingly, embodiments of the present disclosure also provide a computer-readable storage medium storing a computer program. When executed, the computer program can implement the steps that can be performed by an electronic device in the aforementioned method embodiments. Accordingly, embodiments of the present disclosure also provide a computer program product, including a computer program / instructions. When executed by a processor, the computer program / instructions can implement the steps of the method provided in the embodiments of the present disclosure. Those skilled in the art will appreciate that the embodiments of the present disclosure can be provided as methods, systems, or computer program products. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program code. The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, such that execution of the instructions by the processor of the computer or other programmable data processing device produces means for implementing the functions specified in one or more processes in the flowcharts and / or one or more blocks in the block diagrams. These computer program instructions can also be stored in a computer-readable memory capable of directing the computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the functions specified in one or more processes in the flowcharts and / or one or more blocks in the block diagrams. These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. O Memory is an example of computer-readable media. Computer-readable media include both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves. It should also be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, product, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, product, or apparatus. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, product, or apparatus comprising the recited element. The foregoing description is merely an example of the present disclosure and is not intended to limit the present disclosure. Persons skilled in the art will readily appreciate that the present disclosure is susceptible to various modifications and variations. Any modifications, equivalent substitutions, improvements, and the like made within the spirit and principles of the present disclosure are intended to be encompassed by the claims of the present disclosure.

Claims

Claims 1. A question-answering processing method, comprising: Receive inquiry instructions; Acquire target knowledge information matching the query instruction from a plurality of knowledge information in the knowledge base; Obtain a content parsed text corresponding to the target knowledge information, wherein the content parsed text is obtained by performing content parsing on a target document element corresponding to the target knowledge information, wherein the target document element includes a document element in a non-textual mode; input the query instruction and the content parsed text corresponding to the target knowledge information into a large language model, so that the large language model outputs a response result adapted to the query instruction based on the content parsed text corresponding to the target knowledge information.

2. The method according to claim 1, wherein: Obtaining target knowledge information that matches the query instruction from multiple knowledge information in a knowledge base, including: obtaining text representation information of each of the multiple knowledge information in the knowledge base; calculating the matching degree between the query instruction and the text representation information of each of the multiple knowledge information to obtain the matching degree scores between the query instruction and the multiple knowledge information; and determining the target knowledge information that matches the query instruction from the multiple knowledge information according to the matching degree scores between the query instruction and the multiple knowledge information.

3. The method according to claim 2, wherein: Performing a matching calculation on the query instruction and the text representation information of each of the multiple knowledge information to obtain a matching score for each of the query instruction and the multiple knowledge information, including: for any knowledge information among the multiple knowledge information, using multiple recall algorithms to perform a matching calculation on the text representation information of the knowledge information and the query instruction to obtain multiple matching scores; determining weight coefficients corresponding to the multiple recall algorithms based on the modality of the knowledge information and the applicability of the multiple recall algorithms to knowledge information of different modalities; performing a weighted calculation on the multiple matching scores based on the weight coefficients corresponding to the multiple recall algorithms to obtain a matching score for the query instruction and the knowledge information.

4. The method according to claim 2 or 3, wherein: Before obtaining the text representation information of each of the multiple knowledge information in the knowledge base, it also includes: obtaining the multiple document elements from multiple document samples, where the multiple document elements correspond to multiple different modalities; adding the multiple document elements as knowledge information to the knowledge base; performing representation processing on the multiple document elements to obtain the text representation information of each of the multiple document elements, and saving the correspondence between the multiple document elements and their respective text representation information.

5. The method according to claim 4, wherein: Performing characterization processing on the multiple document elements to obtain text characterization information of each of the multiple document elements, including: for any document sample among the multiple document samples, if the document sample includes a text paragraph, obtaining at least one of the name of the document where the text paragraph is located, the paragraph title, the paragraph content, the document creation time, and the editing time as the text characterization information of the text paragraph; if the document sample includes a table in text form, obtaining at least one of the name of the document where the table is located, the table title, the key content in the table, the document creation time, and the editing time as the text characterization information of the table; if the document sample includes a picture, obtaining at least one of the name of the document where the picture is located, the picture title, the question identified from the picture, the description information of the picture, the document creation time, and the editing time as the text characterization information of the picture. The text representation information of the image.

6. The method according to claim 4 or 5, further comprising: According to the modalities corresponding to each of the multiple document elements, the multiple document elements are parsed to obtain content parsed texts corresponding to each of the multiple document elements; multiple question-answer pair samples associated with the multiple document samples are obtained; any question-answer pair sample includes a query instruction sample and a corresponding answer result sample; according to the content parsed texts corresponding to each of the multiple document elements and the multiple question-answer pair samples, multiple sample pairs are obtained; and according to the multiple sample pairs, the large language model is trained.

7. The method according to claim 6, wherein: Parsing the multiple document elements according to the modalities corresponding to the document elements in the multiple document samples to obtain content parsed text corresponding to the multiple document elements includes at least one of the following: for any document sample among the multiple document samples, if the document sample includes a text paragraph, segmenting the text paragraph to obtain multiple text blocks as the content parsed text corresponding to the text paragraph; if the document sample includes a table in text form, identifying the text content and the logical relationship between cells in the text-form table, and encoding according to the identified text content and logical relationship to obtain encoding information as the content parsed text corresponding to the text-form table; if the document sample includes an image, and the image includes a table in image form, identifying the text content and the positions of cell vertices in the image-form table, determining the logical relationship between cells in the image-form table according to the identified cell vertices, and encoding according to the identified text content and logical relationship to obtain encoding information as the content parsed text corresponding to the image-form table; if the document sample includes an image, and the image does not include a table, Then, text is extracted from the image or text description information is generated according to the image as the content parsed text corresponding to the image.

8. The method according to claim 6 or 7, wherein: Acquiring a plurality of question-answer pair samples associated with the plurality of document samples includes: inputting the plurality of document samples and preset prompt words into a question-answering model, so that the question-answering model generates the plurality of question-answer pair samples according to text content in the plurality of document samples under the instruction of the prompt words.

9. The method according to any one of claims 6 to 8, wherein: According to the content parsed texts corresponding to each of the multiple document elements and the multiple question-answer pair samples, multiple sample pairs are obtained, including: for any query instruction sample in the multiple question-answer pair samples, a set recall algorithm is used to determine a target document sample that matches the query instruction sample from the multiple document samples; according to the document elements contained in the target document sample, a document element that is adapted to the query instruction sample is determined; the document element that is adapted to the query instruction sample is used as the knowledge information corresponding to the query instruction, and according to the content parsed text of the question-answer pair sample to which the query instruction belongs and the knowledge information corresponding to the query instruction, a sample pair corresponding to the query instruction sample is obtained.

10. The method according to claim 9, wherein: The document elements adapted to the query instruction sample include at least two document elements of different modalities.

11. An electronic device, wherein include: memory and processor; The memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: perform the steps in the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, it can implement the question-answering processing method described in any one of claims 1 to 10.

13. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the question-answering processing method according to any one of claims 1 to 10 can be implemented. 16

Images