Multimodal hybrid retrieval augmented generation method and apparatus, device, medium, and product

Abstract

The application discloses a multimodal hybrid retrieval enhancement generation method, device, equipment, medium and product, and relates to the technical field of retrieval enhancement generation. The method is to first perform multimodal query intention analysis on the original query statement of a user to obtain query intention weights of each mode, then, for each mode, the required calculation resources and / or priority levels for retrieval are positively allocated according to the corresponding query intention weights, and independent retrieval is performed based on the corresponding allocation results to obtain corresponding retrieval results, then, all retrieval results are subjected to multimodal information fusion and sorting processing to obtain a multimodal content list, finally, the list is integrated into a prompt word and imported into a large language model or a large multimodal model to output a multimodal answer, so that the leap from passive and rigid blind retrieval to active and flexible intelligent decision-making can be realized, the response efficiency, answer accuracy and user experience are significantly improved, and resources are saved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of retrieval augmented generation, and particularly relates to a multi-modal mixed retrieval augmented generation method, device, equipment, medium and product. BACKGROUND

[0002] RAG (Retrieval-Augmented Generation) is a technical architecture combining retrieval technology and generation model, aiming to enhance the generation capability of large language models by dynamically introducing external knowledge base. Specifically, when the model needs to generate text or answer questions, it will first retrieve relevant information from a large document set, and then use these retrieved information to guide the generation of text, thereby improving the quality and accuracy of text generation or question answering.

[0003] At present, the existing standard multi-modal RAG technology mainly uniformly encodes data of different modalities (such as text, image and video, etc.) into the same or multiple vector spaces, then encodes the query when the user initiates the query, and finds the data block (whether it is a text segment or a picture) closest to the query vector in the vector database, and finally provides the retrieved multi-modal information to the large language model (Large Language Model, LLM for short) or large multi-modal model (Large Multimodal Models, LMM for short) to generate the final answer. This standard multi-modal RAG technology is widely used in many online customer service systems to provide more comprehensive answers than pure text. For example, when asked about the "return process", the online customer service system will directly send a link or picture containing a flowchart.

[0004] However, the above-mentioned standard multi-modal RAG technology has the following technical defects in actual application:

[0005] (1) Blind retrieval and resource waste, that is, when processing queries, it often treats all modalities "the same", so it may perform a global search in the database of all modalities, or simply calculate the similarity between the text query and all modalities data, which is a "blind search" behavior, lacking search targeting, or using a simple splicing method, resulting in waste of computing resources and response delay;

[0006] (2) Rigid intent understanding, that is, it is completely "rigid" when processing queries, because it depends on the vectorization of customer questions, then recalls the pre-stored knowledge in the vector database, or relies on fixed keyword rules to match modalities, so it cannot understand complex, ambiguous or novel user intents, resulting in that if the user's question method changes slightly, it may not correctly match the best modality, so it lacks generalization and reasoning ability;

[0007] (3) The answer fusion is rough, that is, due to the simple listing of the retrieved multi-modal information, the lack of organic organization and sorting, the stereoscopic answer cannot be formed with logical coherence and clear primary and secondary, resulting in poor user experience.

[0008] In summary, how to improve the multi-modal hybrid retrieval enhancement generation scheme to overcome any of the above defects is a subject that the skilled in the art needs to study. SUMMARY

[0009] The purpose of the present application is to provide a multi-modal hybrid retrieval enhancement generation method, device, computer equipment, computer readable storage medium and computer program product, to solve the problems of blind retrieval, resource waste, rigid intent understanding and / or rough answer fusion existing in the existing standard multi-modal RAG technology.

[0010] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0011] In a first aspect, a multi-modal hybrid retrieval enhancement generation method is provided, comprising:

[0012] Obtaining an original query sentence of a user;

[0013] Performing multi-modal query intent analysis on the original query sentence to obtain query intent weights of each modality in the multi-modal, wherein the multi-modal includes text, image, video and / or structured data;

[0014] For each modality, the corresponding required computing resources for retrieval and / or the priority level used for retrieval are positively allocated according to the corresponding query intent weight, and based on the corresponding allocation result, the original query sentence is used to perform retrieval in the corresponding database to obtain the corresponding retrieval result;

[0015] The retrieval results of each modality are subjected to multimodal information fusion and sorting processing to obtain a multimodal content list sorted according to query intent weight, query relevance, information quality, and / or information redundancy and complementarity. Specifically, this includes: for each retrieved information in the retrieval results of each modality, the original query statement and corresponding information are converted into vectors using a cross-modal encoder; based on the two converted vectors, the query relevance score between the corresponding information and the original query statement is calculated; and the information quality score of the corresponding information is obtained through information quality analysis. For each pair of retrieved information in the retrieval results of each modality, the corresponding information redundancy and complementarity relationship is analyzed as follows: if one of the two corresponding pieces of information... If text / structured data information describes the content of another piece of information, then the ranking priority of the text / structured data information is reduced, or the text / structured data information is marked as supplementary description of the other information and the ranking priority of the other information is increased, and the other information is associated with the text / structured data information, wherein the other information is image / video information; the query intent weights of each modality, the query relevance scores and information quality scores of each retrieved information, and the information redundancy and complementarity relationships of each pair of retrieved information are imported into a pre-trained learning ranking model, and a multimodal content list is output, which is ranked according to query intent weight, query relevance, information quality, and information redundancy and complementarity of each modality;

[0016] The multimodal content list is integrated into prompt words, and the prompt words are imported into a large language model or a large multimodal model to output a multimodal answer that corresponds to the original query statement and contains text, images, videos and / or structured data;

[0017] The multimodal answer is transmitted to the human-computer interaction interface and presented to the user.

[0018] Based on the above-mentioned invention, a novel scheme for enhancing multimodal hybrid retrieval generation based on multimodal query intent analysis and multimodal information fusion and ranking processing is provided. First, multimodal query intent analysis is performed on the user's original query statement to obtain the query intent weights of each modality. Then, for each modality, the corresponding computational resources required for retrieval and / or the priority level used for retrieval are allocated according to the corresponding query intent weights. Independent retrieval is then performed based on the allocation results to obtain the corresponding retrieval results. Next, multimodal information fusion and ranking processing is performed on all retrieval results to obtain a multimodal content list sorted according to query intent weight, query relevance, information quality, and / or information redundancy and complementarity. Finally, the list is integrated into prompt words and imported into a large language model or a large multimodal model to output the multimodal answer. This achieves a leap from passive, rigid, and blind retrieval to proactive, flexible, and intelligent decision-making, significantly improving the system's response efficiency, answer accuracy, and user experience, while saving resources and retrieval costs, facilitating practical application and promotion.

[0019] In one possible design, multimodal query intent analysis is performed on the original query statement to obtain the query intent weights of each modality in the multimodal analysis, including:

[0020] The original query statement is preprocessed to obtain a new query statement, wherein the preprocessing includes stop word removal and / or spelling error correction.

[0021] The new query statement is encoded into a semantic vector using a pre-trained language model.

[0022] Entity recognition technology is used to identify the target entity from the new query statement;

[0023] Keyword extraction technology is applied to extract intent-indicating query keywords from the new query statement;

[0024] The semantic vector, the target entity, and the query keywords are imported into a pre-trained multi-label classification model, and the query intent weights of each modality in the multimodal model are output. The multimodal model includes text, images, videos, and / or structured data.

[0025] In one possible design, for each modality, the original query statement is used to perform a retrieval in the corresponding database to obtain the corresponding retrieval results, including:

[0026] Based on hybrid retrieval technology and combining keyword matching and vector similarity search, the original query statement is used to perform a retrieval in the text database to obtain text retrieval results;

[0027] And / or, based on hybrid retrieval technology and combining keyword matching and vector similarity search, the original query statement is used to perform a retrieval in the structured database to obtain structured data retrieval results;

[0028] And / or, the original query statement is converted into a vector by a cross-modal encoder, and then the image vector most similar to the vector is searched in the image vector database to obtain the image search results;

[0029] And / or, the original query statement is transformed into a vector by a cross-modal encoder, and then the video vector most similar to the vector is searched in the video vector database to obtain the video search results.

[0030] In one possible design, when the learning ranking model has the ability to suggest the layout of retrieved information based on query intent weight, query relevance, information quality, and / or information redundancy and complementarity, at least one piece of retrieved information in the multimodal content list is marked with a layout suggestion.

[0031] In one possible design, transmitting the multimodal answer to the human-computer interaction interface and presenting it to the user includes:

[0032] The multimodal answer is then transmitted to the human-computer interaction interface.

[0033] The multimodal answers are presented to the user in a rich text or card layout on the human-computer interaction interface.

[0034] Secondly, a multimodal hybrid retrieval enhancement generation device is provided, including a query statement acquisition unit, a query intent analysis unit, a retrieval allocation execution unit, a fusion sorting processing unit, a multimodal answer generation unit, and an answer transmission and presentation unit;

[0035] The query statement acquisition unit is used to acquire the user's original query statement;

[0036] The query intent analysis unit is communicatively connected to the query statement acquisition unit and is used to perform multimodal query intent analysis on the original query statement to obtain the query intent weight of each modality in the multimodal data. The multimodal data includes text, images, videos and / or structured data.

[0037] The retrieval allocation execution unit is communicatively connected to the query intent analysis unit and the query statement acquisition unit, respectively. It is used to allocate the corresponding retrieval computing resources and / or retrieval priority levels according to the corresponding query intent weight for each modality, and to execute the retrieval in the corresponding database using the original query statement based on the corresponding allocation results to obtain the corresponding retrieval results.

[0038] The fusion and sorting processing unit, communicatively connected to the retrieval allocation execution unit, is used to perform multimodal information fusion and sorting processing on the retrieval results of each modality, obtaining a multimodal content list sorted according to query intent weight, query relevance, information quality, and / or information redundancy and complementarity. Specifically, this includes: for each retrieved information in the retrieval results of each modality, converting the original query statement and corresponding information into vectors using a cross-modal encoder; calculating the query relevance score between the corresponding information and the original query statement based on the two converted vectors; and obtaining the information quality score of the corresponding information through information quality analysis. For each pair of retrieved information in the retrieval results of each modality, analyzing the corresponding information redundancy and complementarity relationships as follows: If one of the two pieces of information is text / structured data and describes the content of the other piece of information, then the ranking priority of the text / structured data is reduced, or the text / structured data is marked as supplementary description of the other piece of information, and the ranking priority of the other piece of information is increased, and the other piece of information is associated with the text / structured data, wherein the other piece of information is image / video information; the query intent weights of each modality, the query relevance scores and information quality scores of each retrieved piece of information, and the information redundancy and complementarity relationships of each pair of retrieved pieces of information are imported into a pre-trained learning ranking model, and a multimodal content list is output, which is ranked according to query intent weight, query relevance, information quality, and information redundancy and complementarity of each modality;

[0039] The multimodal answer generation unit is communicatively connected to the fusion and sorting processing unit, and is used to integrate the multimodal content list into prompt words, and import the prompt words into a large language model or a large multimodal model to output a multimodal answer that corresponds to the original query statement and contains text, images, videos and / or structured data;

[0040] The answer transmission and presentation unit is communicatively connected to the multimodal answer generation unit and is used to transmit the multimodal answer to the human-computer interaction interface and present it to the user.

[0041] Thirdly, the present invention provides a computer device comprising a storage module, a processing module, and a transceiver module connected in sequence for communication, wherein the storage module is used to store a computer program, the transceiver module is used to send and receive messages, and the processing module is used to read the computer program and execute the multimodal hybrid retrieval enhancement generation method as described in the first aspect or any possible design in the first aspect.

[0042] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the multimodal hybrid retrieval enhancement generation method as described in the first aspect or any possible design within the first aspect.

[0043] Fifthly, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implement the multimodal hybrid retrieval enhancement generation method as described in the first aspect or any possible design in the first aspect.

[0044] The beneficial effects of the above scheme are:

[0045] (1) This invention creatively provides a new scheme for multimodal hybrid retrieval enhancement based on multimodal query intent analysis and multimodal information fusion and sorting processing. First, multimodal query intent analysis is performed on the user's original query statement to obtain the query intent weight of each modality. Then, for each modality, the corresponding retrieval computing resources and / or retrieval priority are positively correlated according to the corresponding query intent weight. Based on the corresponding allocation results, independent retrieval is performed to obtain the corresponding retrieval results. Then, multimodal information fusion and sorting processing is performed on all retrieval results to obtain a multimodal content list sorted according to query intent weight, query relevance, information quality and / or information redundancy and complementarity. Finally, the list is integrated into prompt words and imported into a large language model or a large multimodal model to output multimodal answers. In this way, a leap from passive, rigid and blind retrieval to active, flexible and intelligent decision-making can be achieved, significantly improving the system's response efficiency, answer accuracy and user experience, and saving resources and retrieval costs.

[0046] (2) It can also greatly improve the effectiveness and intuitiveness of the answer by matching the most suitable data modality for the query question, thereby increasing the one-time solution rate of the question;

[0047] (3) It can also avoid unnecessary global searches of all modal databases, reduce computational resource consumption, and speed up response time;

[0048] (4) It can also provide a comprehensive answer with pictures and text, or even videos, just like an experienced customer service expert, far exceeding pure text customer service;

[0049] (5) This solution is not simply an addition to multimodal retrieval, but introduces an "intelligent brain" - namely "multimodal query intent analysis". This brain can make the optimal retrieval decision before retrieval, solve the defects of "blind retrieval" and "rigid rules" in the existing technology, and through "multimodal information fusion and sorting processing", it can also perform refined "post-processing" on the retrieved information, solve the problems of chaotic information presentation and poor user experience in the existing technology, and facilitate practical application and promotion. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the multimodal hybrid retrieval enhancement generation method provided in an embodiment of this application.

[0052] Figure 2 This is a schematic diagram of the structure of the multimodal hybrid retrieval enhancement generation device provided in the embodiments of this application.

[0053] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these embodiments without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0055] It should be understood that although the terms "first" and "second", etc., may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object may be referred to as the second object, and similarly, the second object may be referred to as the first object, without departing from the scope of the exemplary embodiments of the invention.

[0056] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Another example is A, B and / or C, which can mean that any one of A, B, and C or any combination thereof exists. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone or A and B exist simultaneously. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0057] Example

[0058] like Figure 1 As shown, the multimodal hybrid retrieval enhancement generation method provided in the first aspect of this embodiment can be executed, but is not limited to, by a computer device with certain computing resources, such as a server, a personal computer (PC, referring to a multi-purpose computer of a size, price, and performance suitable for personal use; desktop computers, laptops, mini-laptops, tablets, and ultrabooks are all personal computers), a smartphone, a personal digital assistant (PDA), or a wearable device, etc. Figure 1 As shown, the multimodal hybrid retrieval enhancement generation method includes, but is not limited to, the following steps S1 to S6.

[0059] S1. Obtain the user's original query statement.

[0060] In step S1, the original query statement can be obtained through input via a conventional human-computer interaction interface, but is not limited to this. For example, a user can input the text string "Show me a video of the latest Model A drone operating in the mountains and list its main performance parameters" as a complex original query statement that may imply multiple modal requirements.

[0061] S2. Perform multimodal query intent analysis on the original query statement to obtain the query intent weight of each modality in the multimodal analysis, wherein the multimodal analysis includes, but is not limited to, text, images, videos and / or structured data.

[0062] In step S2, the purpose of the multimodal query intent analysis is not to directly generate search instructions, but to first "think" about what the user actually wants and output a weight vector (i.e., a vector composed of the query intent weights of each modality) containing different data modalities and their corresponding priority search strategies, so as to realize the transformation of the user's natural language query into search instructions with differentiated values ​​for different data modalities. Specifically, multimodal query intent analysis is performed on the original query statement to obtain the query intent weights of each modality in the multimodal analysis, including but not limited to the following steps S21 to S25.

[0063] S21. The original query statement is preprocessed to obtain a new query statement, wherein the preprocessing includes, but is not limited to, stop word removal processing and / or spelling error correction processing.

[0064] In step S21, the preprocessing is also known as data cleaning. The specific processes of stop word removal and spelling error correction are existing technologies and will not be described in detail here.

[0065] S22. Encode the new query statement into a semantic vector using a pre-trained language model.

[0066] In step S22, the language model is used to achieve deep semantic understanding of the new query statement in order to obtain the high-dimensional semantic vector. Specifically, the language model can be, but is not limited to, conventionally trained models such as BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on the Transformer architecture proposed by Google in 2018, which revolutionizes the performance of natural language processing tasks through bidirectional contextual understanding mechanisms) or the Wenxin large model.

[0067] S23. Apply entity recognition technology to identify the target entity from the new query statement.

[0068] In step S23, entity recognition technology is a key technology in natural language processing for locating and classifying named entities in text, primarily identifying predefined categories of entities such as person names, place names, and organizations. For example, based on the query example in step S1, specific entities such as "Model A drone" and "mountainous area" can be identified.

[0069] S24. Apply keyword extraction technology to extract intent-indicating query keywords from the new query statement.

[0070] In step S24, the keyword extraction technique is a core task in the field of natural language processing. It mainly extracts representative terms from text through techniques such as statistical features, graph models, and semantic analysis. For example, based on the query example in step S1, keywords such as "take a look," "video," "list," and "parameters" can be extracted. Among them, "take a look" / "video" strongly points to the video modality, and "list" / "parameters" strongly points to the text or structured data modality.

[0071] S25. Import the semantic vector, the target entity, and the query keywords into a pre-trained multi-label classification model, and output the query intent weights of each modality in the multimodal model. The multimodal model includes, but is not limited to, text, images, videos, and / or structured data.

[0072] In step S25, the multi-label classification model can be specifically trained using, but is not limited to, a Transformer-based classifier or a gradient boosting tree model. The sample data required for training can take the following form: model input items containing semantic vectors, query target entities, and query keywords, and model output items containing pre-labeled query intent weights for each modality. The final output of the multi-label classification model is not a single decision, but a weighted list of modalities to represent the query intent weights of each modality. For example, the modality list could be {video: 0.95, structured_data: 0.9, text: 0.8, image: 0.3}. This modality list (i.e., the weight vector) can precisely guide subsequent modality-based retrieval, indicating that video and structured databases should be prioritized for retrieval, followed by text databases, while image databases have the lowest retrieval priority.

[0073] S3. For each modality, allocate the corresponding computational resources required for retrieval and / or the priority level used for retrieval according to the corresponding query intent weight, and based on the corresponding allocation results, use the original query statement to perform retrieval in the corresponding database to obtain the corresponding retrieval results.

[0074] In step S3, the specific allocation strategy is as follows: the higher the query intent weight, the more computational resources are required for the corresponding modality retrieval and / or the higher the priority level of the retrieval; while the lower the query intent weight, the less computational resources are required for the corresponding modality retrieval and / or the lower the priority level of the retrieval. In this way, based on the weight vector, differentiated retrieval tasks can be allocated to different modality databases for retrieval, achieving the purpose of non-blind retrieval and saving resources. Since the retrieval tasks are performed independently in each modal database, the entire process involves heterogeneous retrieval. Specifically, for each modality, the original query statement is used to perform a retrieval in the corresponding database to obtain the corresponding retrieval results. This includes, but is not limited to: using hybrid retrieval technology combined with keyword matching and vector similarity search to perform a retrieval in a text database to obtain text retrieval results; and / or using hybrid retrieval technology combined with keyword matching and vector similarity search to perform a retrieval in a structured database to obtain structured data retrieval results; and / or converting the original query statement into a vector using a cross-modal encoder, and then searching for the most similar image vector in an image vector database to obtain image search results; and / or converting the original query statement into a vector using a cross-modal encoder, and then searching for the most similar video vector in a video vector database to obtain video search results. The hybrid retrieval technology combines the advantages of full-text retrieval and vector retrieval, achieving complementary precise matching and semantic understanding, and aiming to improve the accuracy and comprehensiveness of information retrieval. The keyword matching can, but is not limited to, the existing BM25 (Best Matching 25) algorithm, which calculates the relevance score between the query and the document. The vector similarity search can, but is not limited to, use the cosine similarity calculation formula. The cross-modal encoder can, but is not limited to, use the encoder structure in the CLIP (Contrastive Language-Image Pre-training, a multimodal neural network model developed by OpenAI that achieves cross-modal semantic association between images and text through contrastive learning, supporting tasks such as image classification and retrieval under zero-shot learning) model. Therefore, the specific retrieval task execution process is based on existing technologies and will not be elaborated further. Furthermore, if the query intent weight of a certain modality is lower than a preset threshold (e.g., 0.1), the retrieval of that modality can be skipped.

[0075] S4. Perform multimodal information fusion and sorting processing on the retrieval results of each modality to obtain a multimodal content list sorted according to query intent weight, query relevance, information quality and / or information redundancy and complementarity.

[0076] In step S4, the multimodal information fusion and sorting process is used to integrate scattered information materials (e.g., two video links, five text paragraphs, and one parameter table image) retrieved from different databases into a high-quality draft answer (specifically, this includes filtering, deduplicating, and sorting all information materials, and planning how to best combine them). Specifically, the multimodal information fusion and sorting process is applied to the retrieval results of each modality to obtain a multimodal content list sorted according to query intent weight, query relevance, information quality, and / or information redundancy and complementarity, including but not limited to steps S41 to S43.

[0077] S41. For each piece of information retrieved in the search results of each modality, the original query statement and the corresponding information are converted into vectors by a cross-modal encoder. Based on the two vectors obtained by conversion, the query relevance score between the corresponding information and the original query statement is calculated. Furthermore, the information quality score of the corresponding information is obtained through information quality analysis.

[0078] In step S41, the cross-modal encoder can be implemented using, but is not limited to, the encoder structure in the CLIP model; the query relevance score can be calculated using, but is not limited to, the cosine similarity formula; the specific process of information quality analysis also uses existing technologies, such as first analyzing image clarity or video duration, and then calculating the information quality score based on the analysis results and a preset formula (generally, higher image clarity or longer video duration results in a higher information quality score; lower image clarity or shorter video duration results in a lower information quality score). Furthermore, if the query relevance score of a retrieved piece of information to the original query statement is lower than a preset score threshold, that retrieved information can be filtered out.

[0079] S42. For each pair of retrieved information in the retrieval results of each modality, the corresponding information redundancy and complementarity relationship can be obtained by analyzing, but is not limited to, the following: if one of the corresponding two pieces of information is text / structured data information and describes the content of the other piece of information, then the ranking priority of the text / structured data information is reduced or the text / structured data information is marked as a supplementary description of the other piece of information, and the ranking priority of the other piece of information is increased, and the other piece of information is associated with the text / structured data information, wherein the other piece of information is image / video information.

[0080] S43. The query intent weights of each modality, the query relevance scores and information quality scores of each retrieved information, and the information redundancy and complementarity relationships of each pair of retrieved information are imported into a pre-trained learning ranking model, and a multimodal content list is output, which is sorted according to the query intent weights, query relevance, information quality, and information redundancy and complementarity of each modality.

[0081] In step S43, the Learning-to-Rank (LTR) model is a ranking model automatically optimized using machine learning techniques in fields such as information retrieval, recommendation systems, and natural language processing. Its core is to learn a ranking function through training data, sorting unordered or partially ordered sets of items (such as web pages, products, and news) according to a specific objective to improve user experience. Therefore, it can be routinely trained to enable the learning-to-rank model to rank the retrieval results of each modality according to query intent weight, query relevance, information quality, and information redundancy and complementarity. To provide layout suggestions for subsequent answer generation steps, preferably, when the learning-to-rank model also has the ability to provide layout suggestions for the retrieved information based on query intent weight, query relevance, information quality, and / or information redundancy and complementarity, at least one piece of retrieved information in the multimodal content list is marked with a layout suggestion, for example, for a pair of images and text, it suggests "insert this chart after this text". In addition, any one or any combination of the query intent weights of each modality, the query relevance scores and information quality scores of each retrieved information, and the information redundancy and complementarity relationships of each pair of retrieved information can be imported into another pre-trained learning ranking model to output another multimodal content list that sorts the retrieval results of each modality according to query intent weights, query relevance, information quality, and / or information redundancy and complementarity.

[0082] S5. Integrate the multimodal content list into prompt words, and import the prompt words into a large language model or a large multimodal model to output a multimodal answer that corresponds to the original query statement and contains text, images, videos and / or structured data.

[0083] In step S5, the prompt words can be, for example, "Please generate a fluent and coherent answer based on the following materials: [Video 1], [Text Summary A], [Parameter Chart B]...", so that the large language model or the large multimodal model can "string together" the separate information materials based on these prompt words and generate a final answer containing text descriptions, embedded image / video links, etc. (that is, seamlessly organizing these information materials into a logically coherent and easy-to-understand multimodal answer). Furthermore, the large language model or the large multimodal model are existing models used in standard multimodal RAG technology, and specific details will not be elaborated here.

[0084] S6. The multimodal answer is transmitted to the human-computer interaction interface and presented to the user.

[0085] In step S6, specifically, the multimodal answer is transmitted to the human-computer interaction interface and presented to the user, including but not limited to: transmitting the multimodal answer to the human-computer interaction interface; and presenting the multimodal answer to the user on the human-computer interaction interface in a rich text (a text format that includes formatting, styles, images, links, and other multimedia elements, which has richer layout and display functions compared to plain text) or card layout.

[0086] Therefore, based on the multimodal hybrid retrieval enhancement generation method described in steps S1 to S6 above, a new scheme for multimodal hybrid retrieval enhancement generation based on multimodal query intent analysis and multimodal information fusion and ranking processing is provided. First, multimodal query intent analysis is performed on the user's original query statement to obtain the query intent weights of each modality. Then, for each modality, the corresponding computational resources required for retrieval and / or the priority level used for retrieval are positively correlated according to the corresponding query intent weights. Independent retrieval is then performed based on the corresponding allocation results to obtain the corresponding retrieval results. Next, multimodal information fusion and ranking processing is performed on all retrieval results to obtain a multimodal content list sorted according to query intent weights, query relevance, information quality, and / or information redundancy and complementarity. Finally, the list is integrated into prompt words and imported into a large language model or a large multimodal model to output multimodal answers. This achieves a leap from passive, rigid, and blind retrieval to proactive, flexible, and intelligent decision-making, significantly improving the system's response efficiency, answer accuracy, and user experience, while saving resources and retrieval costs, facilitating practical application and promotion.

[0087] like Figure 2As shown, the second aspect of this embodiment provides a virtual device for implementing the multimodal hybrid retrieval enhancement generation method described in the first aspect, including a query statement acquisition unit, a query intent analysis unit, a retrieval allocation execution unit, a fusion sorting processing unit, a multimodal answer generation unit, and an answer transmission and presentation unit;

[0088] The query statement acquisition unit is used to acquire the user's original query statement;

[0089] The query intent analysis unit is communicatively connected to the query statement acquisition unit and is used to perform multimodal query intent analysis on the original query statement to obtain the query intent weight of each modality in the multimodal data. The multimodal data includes text, images, videos and / or structured data.

[0090] The retrieval allocation execution unit is communicatively connected to the query intent analysis unit and the query statement acquisition unit, respectively. It is used to allocate the corresponding retrieval computing resources and / or retrieval priority levels according to the corresponding query intent weight for each modality, and to execute the retrieval in the corresponding database using the original query statement based on the corresponding allocation results to obtain the corresponding retrieval results.

[0091] The fusion and sorting processing unit, communicatively connected to the retrieval allocation execution unit, is used to perform multimodal information fusion and sorting processing on the retrieval results of each modality, obtaining a multimodal content list sorted according to query intent weight, query relevance, information quality, and / or information redundancy and complementarity. Specifically, this includes: for each retrieved information in the retrieval results of each modality, converting the original query statement and corresponding information into vectors using a cross-modal encoder; calculating the query relevance score between the corresponding information and the original query statement based on the two converted vectors; and obtaining the information quality score of the corresponding information through information quality analysis. For each pair of retrieved information in the retrieval results of each modality, analyzing the corresponding information redundancy and complementarity relationships as follows: If one of the two pieces of information is text / structured data and describes the content of the other piece of information, then the ranking priority of the text / structured data is reduced, or the text / structured data is marked as supplementary description of the other piece of information, and the ranking priority of the other piece of information is increased, and the other piece of information is associated with the text / structured data, wherein the other piece of information is image / video information; the query intent weights of each modality, the query relevance scores and information quality scores of each retrieved piece of information, and the information redundancy and complementarity relationships of each pair of retrieved pieces of information are imported into a pre-trained learning ranking model, and a multimodal content list is output, which is ranked according to query intent weight, query relevance, information quality, and information redundancy and complementarity of each modality;

[0092] The multimodal answer generation unit is communicatively connected to the fusion and sorting processing unit, and is used to integrate the multimodal content list into prompt words, and import the prompt words into a large language model or a large multimodal model to output a multimodal answer that corresponds to the original query statement and contains text, images, videos and / or structured data;

[0093] The answer transmission and presentation unit is communicatively connected to the multimodal answer generation unit and is used to transmit the multimodal answer to the human-computer interaction interface and present it to the user.

[0094] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be found in the multimodal hybrid retrieval enhancement generation method described in the first aspect, and will not be repeated here.

[0095] like Figure 3As shown, the third aspect of this embodiment provides a computer device for executing the multimodal hybrid retrieval enhancement generation method as described in the first aspect. The device includes a storage module, a processing module, and a transceiver module connected in sequence. The storage module stores a computer program, the transceiver module sends and receives messages, and the processing module reads the computer program and executes the multimodal hybrid retrieval enhancement generation method as described in the first aspect. Specifically, the storage module may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; the processing module may, but is not limited to, use a microprocessor of the STM32F105 series. Furthermore, the computer device may also include, but is not limited to, a power supply module, a display screen, and other necessary components.

[0096] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be found in the multimodal hybrid retrieval enhancement generation method described in the first aspect, and will not be repeated here.

[0097] This fourth aspect of the embodiment provides a computer-readable storage medium storing instructions comprising the multimodal hybrid retrieval enhancement generation method as described in the first aspect. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the multimodal hybrid retrieval enhancement generation method as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0098] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be found in the multimodal hybrid retrieval enhancement generation method described in the first aspect, and will not be repeated here.

[0099] This fifth aspect of the embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the multimodal hybrid retrieval enhancement generation method as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0100] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal hybrid retrieval enhancement generation method, characterized in that, include: Obtain the user's original query statement; Multimodal query intent analysis is performed on the original query statement to obtain the query intent weight of each modality in the multimodal analysis, wherein the multimodal analysis includes text, images, videos and / or structured data. For each modality, the corresponding computational resources required for retrieval and / or the priority level used for retrieval are positively correlated with the corresponding query intent weight. Based on the corresponding allocation results, the original query statement is used to perform a retrieval in the corresponding database to obtain the corresponding retrieval results. The retrieval results of each modality are subjected to multimodal information fusion and sorting processing to obtain a multimodal content list sorted according to query intent weight, query relevance, information quality, and / or information redundancy and complementarity. Specifically, this includes: for each retrieved information in the retrieval results of each modality, the original query statement and corresponding information are converted into vectors using a cross-modal encoder; based on the two converted vectors, the query relevance score between the corresponding information and the original query statement is calculated; and the information quality score of the corresponding information is obtained through information quality analysis. For each pair of retrieved information in the retrieval results of each modality, the corresponding information redundancy and complementarity relationship is analyzed as follows: if one of the two corresponding pieces of information... If text / structured data information describes the content of another piece of information, then the ranking priority of the text / structured data information is reduced, or the text / structured data information is marked as supplementary description of the other information and the ranking priority of the other information is increased, and the other information is associated with the text / structured data information, wherein the other information is image / video information; the query intent weights of each modality, the query relevance scores and information quality scores of each retrieved information, and the information redundancy and complementarity relationships of each pair of retrieved information are imported into a pre-trained learning ranking model, and a multimodal content list is output, which is ranked according to query intent weight, query relevance, information quality, and information redundancy and complementarity of each modality; The multimodal content list is integrated into prompt words, and the prompt words are imported into a large language model or a large multimodal model to output a multimodal answer that corresponds to the original query statement and contains text, images, videos and / or structured data; The multimodal answer is transmitted to the human-computer interaction interface and presented to the user.

2. The multimodal hybrid retrieval enhancement generation method according to claim 1, characterized in that, Perform multimodal query intent analysis on the original query statement to obtain the query intent weights of each modality in the multimodal analysis, including: The original query statement is preprocessed to obtain a new query statement, wherein the preprocessing includes stop word removal and / or spelling error correction. The new query statement is encoded into a semantic vector using a pre-trained language model. Entity recognition technology is used to identify the target entity from the new query statement; Keyword extraction technology is applied to extract intent-indicating query keywords from the new query statement; The semantic vector, the target entity, and the query keywords are imported into a pre-trained multi-label classification model, and the query intent weights of each modality in the multimodal model are output. The multimodal model includes text, images, videos, and / or structured data.

3. The multimodal hybrid retrieval enhancement generation method according to claim 1, characterized in that, For each of the aforementioned modalities, the original query statement is used to perform a retrieval in the corresponding database to obtain the corresponding retrieval results, including: Based on hybrid retrieval technology and combining keyword matching and vector similarity search, the original query statement is used to perform a retrieval in the text database to obtain text retrieval results; And / or, based on hybrid retrieval technology and combining keyword matching and vector similarity search, the original query statement is used to perform a retrieval in the structured database to obtain structured data retrieval results; And / or, the original query statement is converted into a vector by a cross-modal encoder, and then the image vector most similar to the vector is searched in the image vector database to obtain the image search results; And / or, the original query statement is transformed into a vector by a cross-modal encoder, and then the video vector most similar to the vector is searched in the video vector database to obtain the video search results.

4. The multimodal hybrid retrieval enhancement generation method according to claim 1, characterized in that, When the learning ranking model has the ability to make layout suggestions for retrieved information based on query intent weight, query relevance, information quality and / or information redundancy and complementarity, at least one piece of retrieved information in the multimodal content list is marked with a layout suggestion.

5. The multimodal hybrid retrieval enhancement generation method according to claim 1, characterized in that, Transmitting the multimodal answer to the human-computer interaction interface and presenting it to the user includes: The multimodal answer is then transmitted to the human-computer interaction interface. The multimodal answers are presented to the user in a rich text or card layout on the human-computer interaction interface.

6. A multimodal hybrid retrieval enhancement generation device, characterized in that, It includes a query statement acquisition unit, a query intent analysis unit, a retrieval allocation and execution unit, a fusion sorting processing unit, a multimodal answer generation unit, and an answer delivery and presentation unit; The query statement acquisition unit is used to acquire the user's original query statement; The query intent analysis unit is communicatively connected to the query statement acquisition unit and is used to perform multimodal query intent analysis on the original query statement to obtain the query intent weight of each modality in the multimodal data. The multimodal data includes text, images, videos and / or structured data. The retrieval allocation execution unit is communicatively connected to the query intent analysis unit and the query statement acquisition unit, respectively. It is used to allocate the corresponding retrieval computing resources and / or retrieval priority levels according to the corresponding query intent weight for each modality, and to execute the retrieval in the corresponding database using the original query statement based on the corresponding allocation results to obtain the corresponding retrieval results. The fusion and sorting processing unit, communicatively connected to the retrieval allocation execution unit, is used to perform multimodal information fusion and sorting processing on the retrieval results of each modality, obtaining a multimodal content list sorted according to query intent weight, query relevance, information quality, and / or information redundancy and complementarity. Specifically, this includes: for each retrieved information in the retrieval results of each modality, converting the original query statement and corresponding information into vectors using a cross-modal encoder; calculating the query relevance score between the corresponding information and the original query statement based on the two converted vectors; and obtaining the information quality score of the corresponding information through information quality analysis. For each pair of retrieved information in the retrieval results of each modality, analyzing the corresponding information redundancy and complementarity relationships as follows: If one of the two pieces of information is text / structured data and describes the content of the other piece of information, then the ranking priority of the text / structured data is reduced, or the text / structured data is marked as supplementary description of the other piece of information, and the ranking priority of the other piece of information is increased, and the other piece of information is associated with the text / structured data, wherein the other piece of information is image / video information; the query intent weights of each modality, the query relevance scores and information quality scores of each retrieved piece of information, and the information redundancy and complementarity relationships of each pair of retrieved pieces of information are imported into a pre-trained learning ranking model, and a multimodal content list is output, which is ranked according to query intent weight, query relevance, information quality, and information redundancy and complementarity of each modality; The multimodal answer generation unit is communicatively connected to the fusion and sorting processing unit, and is used to integrate the multimodal content list into prompt words, and import the prompt words into a large language model or a large multimodal model to output a multimodal answer that corresponds to the original query statement and contains text, images, videos and / or structured data; The answer transmission and presentation unit is communicatively connected to the multimodal answer generation unit and is used to transmit the multimodal answer to the human-computer interaction interface and present it to the user.

7. A computer device, characterized in that, It includes a storage module, a processing module, and a transceiver module that are sequentially connected in communication. The storage module is used to store a computer program, the transceiver module is used to send and receive messages, and the processing module is used to read the computer program and execute the multimodal hybrid retrieval enhancement generation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores instructions that, when executed on a computer, perform the multimodal hybrid retrieval enhancement generation method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the multimodal hybrid retrieval enhancement generation method as described in any one of claims 1 to 5.

Images