Method, device, computer equipment and readable storage medium for processing graphic question and answer

Abstract

The present application relates to a processing method, device, computer equipment, readable storage medium and computer program product for graphic question and answer. The method comprises: obtaining user question data, determining the data type of the user question data; determining the target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type; determining multiple vector representations matching the vector representation from a preset multimodal database according to each vector matching mode; the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modes; determining candidate document data blocks associated with multiple vector representations; determining the target document data block from the candidate document data according to the similarity between each vector representation and the target vector representation; and determining the graphic data corresponding to the user question data according to the target document data block. The present method can output reliable and accurate graphic data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and readable storage medium for processing graphic question and answer. Background Art

[0002] With the development of the Internet and social media, people are increasingly accustomed to obtaining information through multimedia content. An intelligent question-answering system based on common questions is a typical application product, and it is widely used in real life. For example, it can be applied to various fields such as intelligent customer service and network search. When users consult customer service or use search engines to search for questions, the intelligent question-answering system can find answers that match the user's questions from a large number of resources.

[0003] In the related art, the intelligent question answering system outputs the corresponding text expression based on the semantics and intention of the user's question. However, the related art cannot output reliable and accurate graphic and text data for the user's question. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for processing graphic question and answer that can output reliable and accurate graphic data in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for processing graphic question and answer, comprising:

[0006] Acquire user question data and determine the data type of the user question data;

[0007] Determine a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type;

[0008] According to each of the vector matching modalities, multiple vector representations matching the vector representation are determined from a preset multimodal database; wherein the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modalities;

[0009] determining candidate document data blocks associated with a plurality of said vector representations;

[0010] Determining a target document data block from the candidate document data according to the similarity between each of the vector representations and the target vector representation;

[0011] The graphic and text data corresponding to the user question data are determined according to the target document data block.

[0012] In one embodiment, the step of determining a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type includes:

[0013] In the case where the data type is a text type, extracting keywords from the user question data, performing encoding processing according to the keywords to obtain a text vector representation, and determining the text vector representation as a target vector representation;

[0014] Determine that the vector matching modality represented by the target vector includes a text vector modality, a picture recognition text vector modality, and a picture description vector modality.

[0015] In one embodiment, the step of determining a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type includes:

[0016] In the case where the data type is a picture type, if the user question data does not include text, encoding the user question data to obtain a picture vector representation;

[0017] Performing image content summarization processing on the user question data to obtain an image description;

[0018] Encoding the image description to obtain a picture description vector representation, and determining the picture vector representation and the picture description vector representation as a target vector representation of the user question data;

[0019] Determining the target vector representation includes a picture vector modality, a picture recognition text vector modality, and a picture description vector modality.

[0020] In one embodiment, the step of determining a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type includes:

[0021] In the case where the data type is a picture and text type, if the user question data includes text, image encoding processing is performed on the user question data to obtain a picture vector representation;

[0022] Identify the Chinese characters in the image type to obtain image recognition text, encode the image recognition text to obtain a vector representation of the image recognition text;

[0023] Performing image content summarization processing on the user question data to obtain an image description;

[0024] Encoding the image description to obtain a picture description vector representation;

[0025] Determine the picture vector representation, the picture recognition text vector representation and the picture description vector representation as the target vector representation of the user question data;

[0026] Determine that the vector matching modality represented by the target vector includes text vector modality, picture vector modality, picture recognition text vector modality and picture description vector modality.

[0027] In one embodiment, determining the candidate document data blocks associated with the plurality of vector representations comprises:

[0028] identifying a first index value of a plurality of said vector representations;

[0029] Classify the plurality of vector representations according to the first index value to obtain all vector representations belonging to the same second index value;

[0030] Determine a candidate document data block corresponding to the second index value; wherein the second index value is determined according to the first index value.

[0031] In one embodiment, determining the target document data block from the candidate document data blocks according to the similarity between each of the vector representations and the target vector representation comprises:

[0032] According to the similarity between each of the vector representations and the target vector representation, weighting the similarities belonging to the same candidate document data block to obtain the weighted similarity of the candidate document data block;

[0033] Taking values ​​from the weighted similarities in descending order, if the weighted similarity with the largest value is greater than the first preset similarity, determining the candidate document data block corresponding to the weighted similarity as the target document data block;

[0034] The step of determining the graphic and text data corresponding to the user question data according to the target document data block includes:

[0035] The graphic and text data in the target document data block is determined as the graphic and text data corresponding to the user question data.

[0036] In one embodiment, determining the graphic data corresponding to the user question data according to the target document data block includes:

[0037] If the weighted similarity with the largest value is less than or equal to a first preset similarity, and the similarity of the target vector matching mode of the candidate document data block is greater than a second preset similarity, then generating text information corresponding to the user question data through a large language model; wherein the first preset similarity is greater than the second preset similarity;

[0038] According to the image and the text information of the candidate document data block, the image and text data corresponding to the user question data are determined.

[0039] In one embodiment, determining the graphic data corresponding to the user question data according to the target document data block includes:

[0040] If the similarity of the target vector matching mode of the candidate document data block is less than the second preset similarity, determining an associated picture associated with the user question data;

[0041] Based on the associated image, the graphic and text data corresponding to the user question data is determined.

[0042] In a second aspect, the present application also provides a device for processing graphic question and answer, including:

[0043] A data acquisition module, used to acquire user question data and determine the data type of the user question data;

[0044] A data processing module, used to determine a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type;

[0045] A matching module, configured to determine, according to each of the vector matching modalities, a plurality of vector representations matching the vector representation from a preset multimodal database; wherein the multimodal database includes a plurality of document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modalities;

[0046] A data block determination module, used to determine candidate document data blocks associated with the plurality of vector representations;

[0047] Determining a target document data block from the candidate document data according to the similarity between each of the vector representations and the target vector representation;

[0048] The graphic and text data determination module is used to determine the graphic and text data corresponding to the user question data according to the target document data block.

[0049] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0050] Acquire user question data and determine the data type of the user question data;

[0051] Determine a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type;

[0052] According to each of the vector matching modalities, multiple vector representations matching the vector representation are determined from a preset multimodal database; wherein the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modalities;

[0053] determining candidate document data blocks associated with a plurality of said vector representations;

[0054] Determining a target document data block from the candidate document data according to the similarity between each of the vector representations and the target vector representation;

[0055] The graphic and text data corresponding to the user question data are determined according to the target document data block.

[0056] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0057] Acquire user question data and determine the data type of the user question data;

[0058] Determine a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type;

[0059] According to each of the vector matching modalities, multiple vector representations matching the vector representation are determined from a preset multimodal database; wherein the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modalities;

[0060] determining candidate document data blocks associated with a plurality of said vector representations;

[0061] Determining a target document data block from the candidate document data according to the similarity between each of the vector representations and the target vector representation;

[0062] The graphic and text data corresponding to the user question data are determined according to the target document data block.

[0063] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0064] Acquire user question data and determine the data type of the user question data;

[0065] Determine a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type;

[0066] According to each of the vector matching modalities, multiple vector representations matching the vector representation are determined from a preset multimodal database; wherein the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modalities;

[0067] determining candidate document data blocks associated with a plurality of said vector representations;

[0068] Determining a target document data block from the candidate document data according to the similarity between each of the vector representations and the target vector representation;

[0069] The graphic and text data corresponding to the user question data are determined according to the target document data block.

[0070] The above-mentioned processing method, device, computer equipment, computer-readable storage medium and computer program product for image-text question and answer determine the data type of user question data, determine the target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type; according to each vector matching mode, determine multiple vector representations matching the vector representation from a preset multimodal database, that is, by separating the data of a single modality into multiple modalities, recall and retrieval are performed based on a combination of multiple modalities to improve modality utilization, and on this basis, determine the target document data block from the candidate document data according to the similarity between each vector representation and the target vector representation; determine the image-text data corresponding to the user question data according to the target document data block, and according to the similarity calculation, the most matching target document data block can be accurately located from the multimodal database according to the target vector representation and the determined vector matching mode, and determine the image-text data corresponding to the user question data. Compared with the related technology, this method can output corresponding pictures and more related information when outputting text information, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 An application environment diagram of a method for processing picture-text question-answering in an embodiment;

[0073] Figure 2 A schematic diagram of a process flow of a method for processing graphic question and answer in one embodiment;

[0074] Figure 3 is a schematic diagram of a structured information as a trend graph in an embodiment;

[0075] Figure 4 A schematic diagram of a flow chart of a method for determining a candidate document data block in one embodiment;

[0076] Figure 5 A schematic diagram of a flow chart of a method for determining graphic data in one embodiment;

[0077] Figure 6 It is an application schematic diagram of a method for processing picture-text question and answer in one embodiment;

[0078] Figure 7 is a structural block diagram of a device for processing picture-text question and answer in one embodiment;

[0079] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0081] LLM (Large Language Model): A currently popular language model based on generative algorithms such as ChatGPT.

[0082] RAG (Retieve Augmented Generate): An algorithmic model that combines external text for retrieval and superimposes a large language model for understanding and generation.

[0083] VLM (Visual Language Model): can process both image and text types.

[0084] Intelligent question-answering systems based on common questions are widely used in real life. For example, they can be applied to various fields such as intelligent customer service and network search. When users consult customer service or use search engines to search for questions, intelligent question-answering systems can find answers that match user questions from a large number of resources.

[0085] In related technologies, intelligent question answering systems output corresponding text expressions based on the semantics and intentions of the user's questions. However, users do need to see pictures, and how to make a large language model output pictures is a relatively difficult problem.

[0086] In view of the fact that related technologies cannot output reliable and accurate graphic and text data in response to user questions, a method for processing graphic and text question answering is proposed. The target vector representation of the user question data and multiple vector matching modes of the target vector representation are determined according to the data type of the user question data. Based on the vector matching mode, multiple matching vector representations are determined. According to the similarity between each vector representation and the target vector representation, a target document data block is determined from the candidate document data. According to the target document data block, the graphic and text data corresponding to the user question data is determined.

[0087] The processing method of the picture-text question and answer provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal determines the data type of the user's question data by obtaining the user's question data; determines the target vector representation of the user's question data and multiple vector matching modes of the target vector representation according to the data type; according to each vector matching mode, determines multiple vector representations matching the vector representation from a preset multimodal database; wherein the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modes; determines candidate document data blocks associated with multiple vector representations; determines the target document data block from the candidate document data according to the similarity between each vector representation and the target vector representation; determines the graphic data corresponding to the user's question data according to the target document data block.

[0088] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0089] In an exemplary embodiment, Figure 2 As shown, a method for processing picture-text question answering is provided, and the method is applied to Figure 1 The terminal in is taken as an example to illustrate, including the following steps 202 to 206. Among them:

[0090] Step 202: Obtain user question data and determine the data type of the user question data.

[0091] Among them, data types include text type, image type and graphic data.

[0092] Exemplarily, user question data is obtained, the user question data is identified, and the data type of the user question data is obtained.

[0093] Step 204 , determining a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type.

[0094] Among them, the target vector representation can be understood as information used for recall, and recall can be understood as retrieval.

[0095] The number and type of target vector representations corresponding to different data types are different. The number of target vector representations can be one or more. For example, the number of target vector representations corresponding to the text type is one, which can be a text vector representation. The number of target vector representations corresponding to the picture type can be one or more, which can be a picture vector representation, or a picture vector representation and a picture description vector representation.

[0096] The number of target vector representations and the number of vector matching modalities may be such that the number of target vector representations is less than or equal to the number of vector matching modalities. The vector matching modalities corresponding to the text type may include text vector modalities, image recognition text vector modalities, and image description vector modalities. The vector matching modalities corresponding to the image type may include image vector modalities, image recognition text vector modalities, and image description vector modalities.

[0097] Exemplarily, the user question data is layered according to the layered dimensions corresponding to the data type, and the target vector representation corresponding to each layer is obtained, and multiple vector matching modes of the target vector representation of each layer are determined according to the data type. Among them, the correspondence between the data type and the layered dimension of the user question data is predetermined, and the correspondence between the data type and the vector matching mode is predetermined. For example, the data type is a text type, and the layered dimension of the user question data is a text dimension, and only a text vector representation is obtained. The data type is a picture type, and the layered dimensions include images, image text, and image descriptions, and the picture vector representation, the recognition text vector representation, and the picture description vector representation can be obtained accordingly.

[0098] Step 206 : According to each vector matching modality, a plurality of vector representations matching the vector representation are determined from a preset multimodal database.

[0099] Among them, the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modalities. The multimodal database is predetermined, and each document data block in the multimodal database is stored in a preset format. The metadata may also include a document identifier docid, an image path img_path, a text content text, an image recognition text img_ocr_text, and an image description img_description. The vector representations of different modalities include an image vector representation img_emb, an image description vector representation img_description_emb, a recognition text vector representation img_ocr_emb, and a text vector representation text_emb. The preset storage format may be expressed as:

[0100] {docid: 1, text: xxx, text_emb: [], img_path: aa.png, img_emb:[], img_ocr_text: xxx, img_description: xxx, img_ocr_emb: [], img_description_emb:[]}.

[0101] It is understandable that when determining a multimodal database based on the determined document data, the document data needs to be segmented, and the media such as text, title, table, picture and formula in the document need to be segmented, and the segmentation results obtained after segmentation are encoded to obtain the corresponding vector representations, and the vector representations and the content associated with the vector representations are stored in a preset format to obtain a multimodal database. For example, segmenting a document can obtain the segmentation result of the document, and the segmentation result includes a document identifier, an image path img_path, text text associated with the image, an image description img_description, and an image recognition text img_ocr_text. The final storage format can be expressed as: {docid:1, text: xxx, img_path: aa.png, img_ocr_text: xxx, img_description: xxx}. A document can include multiple pictures, and each picture can have a corresponding picture path img_path, text text associated with the picture, an image description img_description, and an image recognition text img_ocr_text.

[0102] For each document segmentation result, perform vector processing on the segmentation result, use the text embedding model (for example, Ada002 model) to vectorize the text to obtain the text vector representation E0, which can also be represented by text_emb; for images, pure image embedding, image embedding based on CLIP / VISTA to obtain the picture vector representation E1, which can be represented by img_emb; if the picture contains text, perform OCR recognition on the picture to obtain the text, and perform image recognition text-ada embedding on the text to obtain E2, img_ocr_emb; for image content summary, use the vlm model, such as gpt4o to summarize the image content, and obtain the picture description vector representation E3, img_description_emb; where the prompt for summarizing the image content can be:

[0103] The specific format prompt at least includes: please summarize the content of the image, and / or, if it contains text, please describe it according to the original text; and / or, if it contains a flowchart, please introduce it step by step; and / or, if it contains a trend chart, please introduce the trend of the topic; and / or, if it contains character information, please introduce who did what and where, and give a detailed introduction to the character's identity information. Furthermore, the number of words in the summary content shall not exceed the preset number of words, for example, the summary content cannot exceed 50 words.

[0104] The three vector representations obtained above can be stored in the vector database in the following format:

[0105] {docid: 1, text: xxx, text_emb: [], img_path: aa.png, img_emb: [], img_ocr_text: xxx, img_description: xxx, img_ocr_emb: [], img_description_emb: []}. It should be noted that the images in the stored documents and the texts matching the images are determined by the shortest distance, or by other existing methods, which will not be elaborated here.

[0106] Exemplarily, according to each vector matching mode, multiple vector representations matching the vector representation are determined from a preset multimodal database. The vector representation here can be determined by determining the similarity between the target vector representation and the vector representation in the multimodal database, and taking the top N in descending order according to the similarity. For example, if the input data is text, then E0 is obtained for the keyword information in the text, and E0 is used for recall. The similarity D0 is calculated after text embedding to perform rough sorting, and the top 10 are selected as candidates. If the input data is a picture and the picture does not contain text, then E1 is used to recall the top 10. This is commonly used in scenarios such as faces and avatars; if the picture contains structured information, such as trend charts, tables, and other specific scenarios or knowledge information that can be parsed, then E1 and E3 are used for recall; otherwise, if the picture contains a large paragraph of text information, then E1 and E2 are used to recall the top 10 respectively; if the input is graphic data, the text and picture can be combined for recall to determine multiple matching vector representations.

[0107] Step 208: Determine candidate document data blocks associated with multiple vector representations.

[0108] It can be understood that the preset multimodal database is stored in a preset format, and each vector representation has a corresponding document identifier. According to the document identifier, it can be determined which vector representations belong to the same document.

[0109] Step 210: Determine a target document data block from the candidate document data according to the similarity between each vector representation and the target vector representation.

[0110] Exemplarily, candidate document data blocks associated with multiple vector representations are determined, and according to the data type of the user's question data, the target vector representation and the vector matching mode are determined. For the candidate document data block, the similarity of the vector representation belonging to the candidate document data block is weightedly calculated to obtain the comprehensive similarity of the candidate document data block, and the target document data block is determined from the candidate document data based on the comprehensive similarity. The determination method can be to determine the target document data block from the candidate document data with the highest comprehensive similarity, or to determine the first k as the target document data block.

[0111] Step 212: determine the graphic and text data corresponding to the user's question data according to the target document data block.

[0112] It can be understood that the target document data block can be understood as the graphic data of the final rag input, and the graphic data corresponding to the user question data is determined based on this input, and the user question data is the original input in the dialogue system.

[0113] The text and image data corresponding to the user's question data are determined based on the target document data block, and the text and image data in the target document data block can be directly output, or the text and image data corresponding to the question data can be output based on the understanding and processing of the target document data block. For example, the search results (e.g., candidate document data blocks) are sorted by similarity. If the top 1 is a type containing pictures, and the similarity is greater than 0.9, the text and image description is directly returned.

[0114] The above-mentioned method for processing image and text questions and answers determines the data type of user question data, determines the target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type; according to each vector matching mode, determines multiple vector representations matching the vector representation from a preset multimodal database, that is, by separating the data of a single modality into multiple modalities, recall and retrieval are performed based on a combination of multiple modalities to improve modality utilization, and on this basis, determines the target document data block from the candidate document data according to the similarity between each vector representation and the target vector representation; determines the image and text data corresponding to the user question data according to the target document data block, and according to the similarity calculation, can accurately locate the most matching target document data block from the multimodal database according to the target vector representation and the determined vector matching mode, and determines the image and text data corresponding to the user question data. Compared with the related technology, this method can output corresponding pictures and more related information when outputting text information, thereby improving the user experience.

[0115] In actual applications, the user question data of the question-answering system has multiple data forms. For different data forms, the input question data needs to be separated into multiple modes and matched according to the matching modes corresponding to each mode. In an exemplary embodiment, the target vector representation of the user question data is determined according to the data type, and the multiple vector matching modes of the target vector representation include the following cases:

[0116] Case 1: When the data type is text type, keywords of the user question data are extracted, encoded according to the keywords to obtain a text vector representation, and the text vector representation is determined as the target vector representation; the vector matching modalities of the target vector representation are determined to include text vector modality, image recognition text vector modality and image description vector modality.

[0117] Among them, the keywords are preset. It can be understood that for user question data of text type, the vectors of the three modes E0, E2 and E3 can be determined respectively according to the above picture type vectorization scheme, and then the similarities D0, D2 and D3 are calculated according to the corresponding data to obtain the corresponding comprehensive similarity -text=0.9×D0+0.06×D2+0.04×D3.

[0118] Case 2: When the data type is a picture type, the user question data is a picture. If the picture does not include text, the user question data is encoded to obtain a picture vector representation, and the picture vector representation is determined as the target vector representation; multiple vector matching modalities of the target vector representation are determined, including a picture vector modality, a picture recognition text vector modality, and a picture description vector modality.

[0119] Case three: If the picture includes text, identify the text type; when the text type is structured information, summarize the content of the picture to obtain an image description; encode the image description to obtain a picture description vector representation; determine the picture vector representation and the picture description vector representation as the target vector representation of the user question data; determine that the target vector representation includes the picture vector modality, the picture recognition text vector modality and the picture description vector modality.

[0120] When the text type is text information, perform text recognition on the image to obtain image recognition text; encode the image recognition text to obtain a recognition text vector representation; determine the image vector representation and the recognition text vector representation as the target vector representation of the user question data; determine that the target vector representation includes the image vector modality, the image recognition text vector modality and the image description vector modality.

[0121] Among them, structured information at least includes trend charts, tables and other specific scenarios or knowledge information that can be parsed, such as Figure 3 As shown, the structured information in an exemplary embodiment is a trend graph. The text information can be understood as a large paragraph of text information included in the picture, and a large paragraph can be understood as the number of characters exceeding the preset number of characters.

[0122] It can be understood that if the input data, that is, the user's question data, is an image and does not contain text, then E1 is used as the recall top 10, and the scenarios used are such as faces and avatars. If the image contains structured information, such as trend charts, tables, and other specific scenarios or knowledge information that can be parsed, then E1 and E3 are used for recall. If the image contains a large amount of text information, then E1 and E2 are used as the recall top 10 respectively.

[0123] For the image type, according to the above-mentioned image data vectorization scheme, the vectors of the three modes E1, E2, and E3 are calculated respectively, and then the similarities D1, D2, and D3 are calculated according to the corresponding data. The corresponding comprehensive similarity can be expressed as:

[0124] Comprehensive similarity -img = 0.5×D1+0.3×D2+0.2×D3. It should be noted that, considering the gap between the similarity of text and image, the similarity of image and text is calculated separately, and the inner product between text vector and image vector is not considered. The similarity between text and text type after image recognition is not considered here, and it is necessary to ensure that different modalities do not interfere with each other.

[0125] Case 4: When the data type is a text and image type, determining that the target vector representation of the user question data includes at least a text vector representation and a picture vector representation; determining that the vector matching modality of the target vector representation includes a text vector modality, a picture vector modality, a picture recognition text vector modality, and a picture description vector modality.

[0126] The above text processing method is used for the text in the user question data, and the above image processing method is used for the user question data, and the corresponding comprehensive similarity can be expressed as:

[0127] Comprehensive similarity IT_Sim=0.6×comprehensive similarity-text+0.4×comprehensive similarity-img.

[0128] It should be noted here that the similarity weights in the above expressions are obtained by testing the top3 recall of a batch of sample data sets for image and text training through hyperparameter search, which are the optimal solutions on the training set and the validation set.

[0129] In the above embodiment, the user question data can be analyzed in a more refined layered manner, and the data of a single modality can be separated into multiple modalities as much as possible, and then combined with recall and retrieval to improve the modality utilization rate.

[0130] In an exemplary embodiment, Figure 4 As shown, a method for determining a candidate document data block is provided, comprising steps 402 to 406, wherein:

[0131] Step 402, identifying a first index value represented by a plurality of vectors.

[0132] The first index value may be understood as a document identifier, and the first index value corresponding to each vector representation is predetermined.

[0133] Step 404: classify the multiple vector representations according to the first index value to obtain all vector representations belonging to the same second index value.

[0134] Step 406: determine a candidate document data block corresponding to a second index value; wherein the second index value is determined based on the first index value.

[0135] In the above method, indexes are established according to the modality, and the recall algorithm of multimodal similarity is solved by alignment weighting. At the same time, the multimodal data is sorted to obtain the final rag input image and text data, and then the accurate multimodal output result can be determined. The multimodality here includes at least two modalities: picture and text.

[0136] Based on the final rag input image and text data, it is necessary to select the appropriate output time of the image type data according to the similarity as the final multimodal output result. Figure 5 As shown, a method for determining graphic data is provided, including steps 502 to 518, wherein:

[0137] Step 502 : weighting the similarities belonging to the same candidate document data block according to the similarity between each vector representation and the target vector representation, to obtain the weighted similarity of the candidate document data block.

[0138] The calculation method of the weighted similarity can be determined based on the calculation methods of the above-mentioned different input data types, which will not be elaborated here.

[0139] Step 504 , determining whether the weighted similarity is greater than a first preset similarity, if so, executing step 506 , otherwise, executing step 510 .

[0140] The first preset similarity may be, but is not limited to, 0.9.

[0141] Step 506 , taking values ​​from the weighted similarities in descending order, and if the weighted similarity with the largest value is greater than the first preset similarity, determining the candidate document data block corresponding to the weighted similarity as the target document data block.

[0142] Step 508: determine the graphic and text data in the target document data block as the graphic and text data corresponding to the user's question data.

[0143] For example, according to the similarity between each vector representation and the target vector representation, the similarities belonging to the same candidate document data block are weighted to obtain the weighted similarity of the candidate document data block, and the candidate document data blocks are sorted based on the weighted similarity. If the top 1 in the sorting result is a type containing pictures, and the similarity between this result and the query is greater than 0.9 (i.e. very similar), then the system will directly return the picture and text description of this result. This means that the user will get a highly relevant picture and the text description related to it.

[0144] Step 510 , determining whether the similarity of the target vector matching mode is greater than a second preset similarity, if so, executing step 512 , otherwise, executing step 518 .

[0145] The second preset similarity may be, but is not limited to, 0.8, and the target vector matching modality includes at least one of a picture vector modality, a picture recognition text vector modality, and a picture description vector modality. Here, the input data is of a picture type or a picture and text type.

[0146] Step 512: If the weighted similarity with the largest value is less than or equal to the first preset similarity, and the similarity of the target vector matching mode of the candidate document data block is greater than the second preset similarity, then text information corresponding to the user question data is generated through the large language model; wherein the first preset similarity is greater than the second preset similarity.

[0147] Step 514: Determine the graphic and text data corresponding to the user's question data based on the image and text information of the candidate document data block.

[0148] Exemplarily, the candidate document data blocks are sorted based on weighted similarity. If the weighted similarity with the largest value is less than or equal to the first preset similarity, the target vector matching mode of the candidate document data block needs to be judged. If the similarity of any one of the corresponding image vector mode, image recognition text vector mode and image description vector mode is greater than the second preset similarity, such as 0.8, the text output of the large language model and the image in the candidate document data block are used to determine the final output image and text data. Furthermore, the final output image can be determined by the topk images in the candidate document data block with similarity, and the number of images can be unlimited here.

[0149] Step 516: If the similarity of the target vector matching mode of the candidate document data block is less than a second preset similarity, then determine an associated picture associated with the user question data.

[0150] Step 518: Determine the graphic and text data corresponding to the user's question data based on the associated image.

[0151] Exemplarily, if the candidate document data blocks are sorted based on weighted similarity, if the weighted similarity with the largest value is less than or equal to a first preset similarity, such as 0.9, and the image vector modality, image recognition text vector modality and image description vector modality similarities corresponding to the candidate document data block are all less than a second preset similarity, such as 0.8, the vlm model is used to determine whether the image in the candidate document data block is relevant to the input question. If so, the relevant image can be brought out to the answer, and the large language model can reorganize the format of the answer according to the image and text answer to obtain the final image and text data.

[0152] In this embodiment, indexes are established according to the modalities, and the recall algorithm of multimodal similarity is solved by alignment weighting. At the same time, the multimodal data is sorted to obtain the final rag input image and text data. On the basis of determining the final rag input image and text data, it is necessary to select the appropriate output time of the image type data according to the similarity. As the final multimodal output result, the text question can be converted into a picture and text question and answer.

[0153] In an exemplary embodiment, a method for processing a picture-text question and answer is provided, comprising the following steps:

[0154] Acquire user question data and determine the data type of the user question data; determine the target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type; determine multiple vector representations matching the vector representation from a preset multimodal database according to each vector matching mode; wherein the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modes; identify first index values ​​of multiple vector representations; classify the multiple vector representations according to the first index value to obtain all vector representations belonging to the same second index value; determine the candidate document data block corresponding to the second index value; wherein the second index value is determined according to the first index value.

[0155] According to the similarity between each vector representation and the target vector representation, the similarities belonging to the same candidate document data block are weighted to obtain the weighted similarity of the candidate document data block; values ​​are taken from the weighted similarities in descending order, and if the weighted similarity with the largest value is greater than the first preset similarity, the candidate document data block corresponding to the weighted similarity is determined as the target document data block; the graphic data in the target document data block is determined as the graphic data corresponding to the user question data. The graphic data corresponding to the user question data is determined according to the target document data block.

[0156] If the weighted similarity with the largest value is less than or equal to the first preset similarity, and the similarity of the target vector matching mode of the candidate document data block is greater than the second preset similarity, then text information corresponding to the user question data is generated through the large language model; wherein the first preset similarity is greater than the second preset similarity; and based on the image and text information of the target document data block, the graphic and text data corresponding to the user question data is determined.

[0157] If the similarity of the target vector matching mode of the candidate document data block is less than the second preset similarity, an associated image associated with the user question data is determined; based on the associated image, graphic and text data corresponding to the user question data is determined.

[0158] It is understandable that the specific method of this embodiment can be implemented in the above-mentioned limited manner, which will not be described in detail here. Further, the processing method of picture and text question and answer can be applied to internal sales and operation students using pictures and text to ask questions, such as product page errors, etc., and the problem can be directly located through screenshots.

[0159] The following are application scenarios of the processing method of picture and text question and answer, such as Figure 6 As shown, when it is detected that the knowledge base increment has been updated, special documents such as tables / PPT / Word in the incremental data are obtained for segmentation to obtain the segmentation results, and the segmentation results are vectorized according to the Eembedding vector embedding algorithm to obtain document data blocks associated with each document. The document data blocks include document identifier docid, image path img_path, text content text, image recognition text img_ocr_text and image description img_description. The vector representations of different modalities include image vector representation img_emb, image description vector representation img_description_emb, recognition text vector representation img_ocr_emb and text vector representation text_emb, etc., and the vector processing results are stored in the vector database according to the preset format, that is, the preset multimodal database is obtained. The knowledge base includes the DingTalk cloud disk knowledge base and the document knowledge base (for example, the DZZ-CMS document database).

[0160] Obtain user question data, determine keywords / vectors for searching the user question data, and execute the steps of the above-mentioned image-text question-answering processing method, wherein, when the similarity is greater than 0.9, the corresponding target document data can be directly output as the result, and when the similarity is less than or equal to 0.9, it is processed by the large language model. If the similarity of any dimension D1 / D2 / D3 is greater than 0.8: then the text output of the large language model and the image corresponding to the similarity of any dimension D1 / D2 / D3 greater than 0.8 are used as the final output to obtain the image-text data; if all dimensions D1 / D2 / D3 are less than 0.8, then the vlm model is used to determine whether the image and the question are related. If they are related, they can be brought out to the answer, and the large language model is allowed to reorganize the format of the answer according to the image and text answer. The final output result can be output in the markup language (markdown) format supported by DingTalk.

[0161] The above-mentioned method for processing image and text questions and answers determines the data type of user question data, determines the target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type; according to each vector matching mode, determines multiple vector representations matching the vector representation from a preset multimodal database, that is, by separating the data of a single modality into multiple modalities, recall and retrieval are performed based on a combination of multiple modalities to improve modality utilization, and on this basis, determines the target document data block from the candidate document data according to the similarity between each vector representation and the target vector representation; determines the image and text data corresponding to the user question data according to the target document data block, and according to the similarity calculation, can accurately locate the most matching target document data block from the multimodal database according to the target vector representation and the determined vector matching mode, and determines the image and text data corresponding to the user question data. Compared with the related technology, this method can output corresponding pictures and more related information when outputting text information, thereby improving the user experience.

[0162] It is understandable that by effectively encoding the text and image information, the problem of inputting text / images and outputting text / images in the dialogue system is solved. An automatic routing mechanism is established for multimodal input, and the knowledge base information is indexed according to the modality. The recall algorithm of multimodal similarity is solved through alignment weighting, and the multimodal data is sorted to obtain the final rag input text and image data. Finally, the appropriate output time of the image type data is selected according to the similarity as the final multimodal output result.

[0163] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiment of the present application also provides a device for processing graphic question and answer for implementing the graphic question and answer processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more graphic question and answer processing devices provided below can refer to the limitations of the graphic question and answer processing method above, and will not be repeated here.

[0165] In an exemplary embodiment, Figure 7 As shown, a processing device for image-text question answering is provided, including: a data acquisition module 702, a data processing module 704, a matching module 706, a data block determination module 708 and an image-text data determination module 710, wherein:

[0166] The data acquisition module 702 is used to acquire user question data and determine the data type of the user question data.

[0167] The data processing module 704 is used to determine the target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type.

[0168] The matching module 706 is used to determine multiple vector representations that match the vector representation from a preset multimodal database according to each vector matching modality; wherein the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modalities.

[0169] The data block determination module 708 is configured to determine candidate document data blocks associated with the plurality of vector representations.

[0170] According to the similarity between each vector representation and the target vector representation, a target document data block is determined from the candidate document data.

[0171] The graphic and text data determination module 710 is used to determine the graphic and text data corresponding to the user's question data according to the target document data block.

[0172] The above-mentioned image-text question and answer processing device determines the data type of the user's question data, determines the target vector representation of the user's question data, and multiple vector matching modes of the target vector representation according to the data type; according to each vector matching mode, determines multiple vector representations matching the vector representation from a preset multimodal database, that is, by separating the data of a single modality into multiple modalities, recall and retrieval are performed based on a combination of multiple modalities to improve the modality utilization rate, and on this basis, determines the target document data block from the candidate document data according to the similarity between each vector representation and the target vector representation; determines the image-text data corresponding to the user's question data according to the target document data block, and according to the similarity calculation, can accurately locate the most matching target document data block from the multimodal database according to the target vector representation and the determined vector matching mode, and determines the image-text data corresponding to the user's question data. Compared with the related technology, this method can output corresponding pictures when outputting text information, output more related information, and improve the user experience.

[0173] In an exemplary embodiment, the data processing module 704 is further used to extract keywords from the user question data when the data type is a text type, perform encoding processing according to the keywords, obtain a text vector representation, and determine the text vector representation as a target vector representation;

[0174] The vector matching modalities for determining the target vector representation include text vector modality, image recognition text vector modality and image description vector modality.

[0175] In an exemplary embodiment, the data processing module 704 is further configured to, when the data type is a picture type and the user question data is a picture, if the picture does not include text, encode the user question data to obtain a picture vector representation, and determine the picture vector representation as the target vector representation;

[0176] The multiple vector matching modalities for determining the target vector representation include image vector modality, image recognition text vector modality and image description vector modality.

[0177] In an exemplary embodiment, the data processing module 704 is further configured to identify the text type of the text if the image includes text;

[0178] When the text type is structured information, the image is summarized to obtain an image description;

[0179] Encode the image description to obtain a vector representation of the image description;

[0180] Determine the image vector representation and the image description vector representation as the target vector representation of the user question data;

[0181] Determine the target vector representation includes image vector modality, image recognition text vector modality and image description vector modality.

[0182] In an exemplary embodiment, the data processing module 704 is further configured to perform text recognition on the image to obtain image recognition text when the text type is text information;

[0183] Encode the image recognition text to obtain the recognized text vector representation;

[0184] Determine the image vector representation and the recognized text vector representation as the target vector representation of the user question data;

[0185] Determine the target vector representation includes image vector modality, image recognition text vector modality and image description vector modality.

[0186] In an exemplary embodiment, the data processing module 704 is further used to determine that the target vector representation of the user question data at least includes a text vector representation and a picture vector representation when the data type is a picture-text type;

[0187] The vector matching modalities for determining the target vector representation include text vector modality, image vector modality, image recognition text vector modality and image description vector modality.

[0188] In an exemplary embodiment, the data block determination module 708 is further configured to identify a first index value of the plurality of vector representations;

[0189] Classify the multiple vector representations according to the first index value to obtain all vector representations belonging to the same second index value;

[0190] Determine a candidate document data block corresponding to a second index value; wherein the second index value is determined based on the first index value.

[0191] In an exemplary embodiment, the data block determination module 708 is further configured to weight the similarities of the candidate document data blocks belonging to the same candidate document data block according to the similarities between each vector representation and the target vector representation, to obtain the weighted similarities of the candidate document data blocks;

[0192] Taking values ​​from the weighted similarities in descending order, if the weighted similarity with the largest value is greater than the first preset similarity, determining the candidate document data block corresponding to the weighted similarity as the target document data block;

[0193] In an exemplary embodiment, the graphic data determination module 710 is further configured to determine the graphic data in the target document data block as graphic data corresponding to the user question data.

[0194] In an exemplary embodiment, the graphic data determination module 710 is further configured to generate text information corresponding to the user question data through a large language model if the weighted similarity with the largest value is less than or equal to a first preset similarity and the similarity of the target vector matching mode of the candidate document data block is greater than a second preset similarity; wherein the first preset similarity is greater than the second preset similarity;

[0195] According to the image and text information of the target document data block, the image and text data corresponding to the user's question data are determined.

[0196] In an exemplary embodiment, the image and text data determination module 710 is further configured to determine an associated image associated with the user question data if the similarity of the target vector matching modality of the candidate document data block is less than a second preset similarity;

[0197] Based on the associated pictures, the graphic and text data corresponding to the user's question data is determined.

[0198] Each module in the above-mentioned graphic question-and-answer processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0199] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for processing graphic questions and answers is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0200] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0201] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0202] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0203] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0205] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0206] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0207] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing picture and text question and answer, characterized in that: The method comprises: Acquire user question data and determine the data type of the user question data; Determine a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type; According to each of the vector matching modalities, multiple vector representations matching the target vector representation are determined from a preset multimodal database; wherein the multimodal database includes multiple document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modalities; determining candidate document data blocks associated with a plurality of said vector representations; Determine a target document data block from the candidate document data blocks according to the similarity between each of the vector representations and the target vector representation; the target document data block is determined according to the comprehensive similarity of the candidate document data blocks obtained by weighted calculation of the similarities of the vector representations belonging to the same candidate document data block; Determining graphic and text data corresponding to the user question data according to the target document data block; wherein the correspondence between the data type and the hierarchical dimension of the user question data is predetermined, and the correspondence between the data type and the vector matching modality is predetermined; The data types include text type, picture type and picture-text type, and the vector matching modalities corresponding to the text types include text vector modality, picture recognition text vector modality and picture description vector modality; The vector matching modalities corresponding to the picture type include picture vector modality, picture recognition text vector modality and picture description vector modality; The vector matching modalities corresponding to the image and text types include text vector modality, picture vector modality, picture recognition text vector modality and picture description vector modality.

2. The method according to claim 1, characterized in that The step of determining the target vector representation of the user question data according to the data type, and multiple vector matching modes of the target vector representation, includes: In the case where the data type is a text type, extracting keywords from the user question data, performing encoding processing according to the keywords to obtain a text vector representation, and determining the text vector representation as a target vector representation; Determine that the vector matching modality represented by the target vector includes a text vector modality, a picture recognition text vector modality, and a picture description vector modality.

3. The method according to claim 1, characterized in that The step of determining the target vector representation of the user question data according to the data type, and multiple vector matching modes of the target vector representation, includes: In the case where the data type is a picture type, the user question data is a picture, and if the picture does not include text, the user question data is encoded to obtain a picture vector representation, and the picture vector representation is determined as the target vector representation; Determine that the target vector represents a plurality of vector matching modalities including a picture vector modality, a picture recognition text vector modality and a picture description vector modality.

4. The method according to claim 3, characterized in that: The method further comprises: If the image includes text, identifying the text type of the text; When the text type is structured information, performing content summarization processing on the picture to obtain an image description; Encoding the image description to obtain a picture description vector representation; Determine the picture vector representation and the picture description vector representation as the target vector representation of the user question data; Determine that the target vector represents a plurality of vector matching modalities including a picture vector modality, a picture recognition text vector modality and a picture description vector modality.

5. The method according to claim 4, characterized in that The method further comprises: When the text type is text information, performing text recognition on the image to obtain image recognition text; Encoding the image recognition text to obtain a recognition text vector representation; Determine the picture vector representation and the recognized text vector representation as target vector representations of the user question data; Determine that the target vector represents a plurality of vector matching modalities including a picture vector modality, a picture recognition text vector modality and a picture description vector modality.

6. The method according to claim 1, characterized in that The step of determining the target vector representation of the user question data according to the data type, and multiple vector matching modes of the target vector representation, includes: In the case where the data type is a picture and text type, determining that the target vector representation of the user question data at least includes a text vector representation and a picture vector representation; Determine that the vector matching modality represented by the target vector includes text vector modality, picture vector modality, picture recognition text vector modality and picture description vector modality.

7. The method according to any one of claims 1 to 6, characterized in that: The determining of candidate document data blocks associated with the plurality of vector representations comprises: identifying a first index value of a plurality of said vector representations; Classify the plurality of vector representations according to the first index value to obtain all vector representations belonging to the same second index value; Determine a candidate document data block corresponding to the second index value; wherein the second index value is determined according to the first index value.

8. The method according to claim 7, characterized in that Determining the target document data block from the candidate document data blocks according to the similarity between each of the vector representations and the target vector representation comprises: According to the similarity between each of the vector representations and the target vector representation, weighting the similarities belonging to the same candidate document data block to obtain the weighted similarity of the candidate document data block; Taking values ​​from the weighted similarities in descending order, if the weighted similarity with the largest value is greater than the first preset similarity, determining the candidate document data block corresponding to the weighted similarity as the target document data block; The step of determining the graphic and text data corresponding to the user question data according to the target document data block includes: The graphic and text data in the target document data block is determined as the graphic and text data corresponding to the user question data.

9. The method according to claim 8, characterized in that The step of determining the graphic and text data corresponding to the user question data according to the target document data block includes: If the weighted similarity with the largest value is less than or equal to a first preset similarity, and the similarity of the target vector matching mode corresponding to the candidate document data block is greater than a second preset similarity, then text information corresponding to the user question data is generated through a large language model; wherein the first preset similarity is greater than the second preset similarity; According to the image and the text information of the candidate document data block, the image and text data corresponding to the user question data are determined.

10. The method according to claim 9, characterized in that The step of determining the graphic and text data corresponding to the user question data according to the target document data block includes: If the similarity of the target vector matching mode of the candidate document data block is less than the second preset similarity, determining an associated picture associated with the user question data; Based on the associated image, the graphic and text data corresponding to the user question data is determined.

11. A device for processing picture and text questions and answers, characterized in that: The device comprises: A data acquisition module, used to acquire user question data and determine the data type of the user question data; A data processing module, used to determine a target vector representation of the user question data and multiple vector matching modes of the target vector representation according to the data type; A matching module, configured to determine, according to each of the vector matching modalities, a plurality of vector representations matching the target vector representation from a preset multimodal database; wherein the multimodal database includes a plurality of document data blocks, each document data block includes metadata, and the metadata includes vector representations of different modalities; A data block determination module, used to determine candidate document data blocks associated with the plurality of vector representations; Determine a target document data block from the candidate document data blocks according to the similarity between each of the vector representations and the target vector representation; the target document data block is determined according to the comprehensive similarity of the candidate document data blocks obtained by weighted calculation of the similarities of the vector representations belonging to the same candidate document data block; A graphic data determination module, used to determine the graphic data corresponding to the user question data according to the target document data block; wherein the correspondence between the data type and the hierarchical dimension of the user question data is predetermined, and the correspondence between the data type and the vector matching mode is predetermined; The data types include text type, picture type and picture-text type, and the vector matching modalities corresponding to the text types include text vector modality, picture recognition text vector modality and picture description vector modality; The vector matching modalities corresponding to the picture type include picture vector modality, picture recognition text vector modality and picture description vector modality; The vector matching modalities corresponding to the image and text types include text vector modality, picture vector modality, picture recognition text vector modality and picture description vector modality.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images