Multi-modal mixed retrieval enhancement generation method and device, equipment, medium and product

Abstract

The invention discloses a multi-modal mixed retrieval enhancement generation method and device, equipment, a medium and a product, and relates to the technical field of retrieval enhancement generation. The method comprises the steps that firstly, multi-mode query intention analysis is conducted on an original query statement of a user to obtain query intention weights of all modes, and then corresponding calculation resources needed by retrieval and / or priority levels used by retrieval are distributed in a positive correlation mode according to the corresponding query intention weights for all the modes. The method comprises the following steps: acquiring a corresponding distribution result, executing independent retrieval based on the corresponding distribution result to obtain a corresponding retrieval result, then carrying out multi-modal information fusion and sorting processing on all the retrieval results to obtain a multi-modal content list, finally integrating the list into a cue word, importing the cue word into a large language model or a large multi-modal model, and outputting to obtain a multi-modal answer. In this way, the jump from passive, rigid and blind retrieval to active, flexible and intelligent decision making can be realized, the response efficiency, the answer accuracy and the user experience are remarkably improved, and resources are saved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of retrieval augmented generation, and particularly relates to a multi-modal mixed retrieval augmented generation method, device, equipment, medium and product. BACKGROUND

[0002] RAG (Retrieval-Augmented Generation) is a technical architecture combining retrieval technology and generation model, aiming to enhance the generation capability of large language models by dynamically introducing external knowledge base. Specifically, when the model needs to generate text or answer questions, it will first retrieve relevant information from a large document set, and then use these retrieved information to guide the generation of text, thereby improving the quality and accuracy of text generation or question answering.

[0003] At present, the existing standard multi-modal RAG technology mainly uniformly encodes data of different modalities (such as text, image and video, etc.) into the same or multiple vector spaces, then encodes the query when the user initiates the query, and finds the data block (whether it is a text segment or a picture) closest to the query vector in the vector database, and finally provides the retrieved multi-modal information to the large language model (Large Language Model, LLM for short) or large multi-modal model (Large Multimodal Models, LMM for short) to generate the final answer. This standard multi-modal RAG technology is widely used in many online customer service systems to provide richer answers than pure text. For example, when asked about the "return process", the online customer service system will directly send a link or picture containing a flowchart.

[0004] However, the above-mentioned standard multi-modal RAG technology has the following technical defects in actual application: (1) Blind retrieval and resource waste, that is, when processing queries, it often treats all modalities "the same", so it may perform a global search in the database of all modalities, or simply calculate the similarity between the text query and all modal data, which is a "blind search" behavior, lacking search targeting, or using a simple splicing method, resulting in waste of computing resources and response delay; (2) Rigid intent understanding, that is, it is completely "rigid" when processing queries, because it depends on the vectorization of customer questions, then recalls the pre-stored knowledge in the vector database, or relies on fixed keyword rules to match modalities, so it cannot understand complex, ambiguous or novel user intents, resulting in that if the user's question method changes slightly, it may not correctly match the best modality, so it lacks generalization and reasoning ability; (3) The answer fusion is rough, that is, due to the simple listing of the retrieved multi-modal information, the lack of organic organization and sorting, the stereoscopic answer cannot be formed to be logically coherent and clear in primary and secondary, and thus the user experience is poor.

[0005] In summary, how to improve the multi-modal hybrid retrieval enhancement generation scheme to overcome any of the above defects is a subject that the person skilled in the art needs to study. SUMMARY

[0006] The purpose of the present application is to provide a multi-modal hybrid retrieval enhancement generation method, device, computer equipment, computer readable storage medium and computer program product, to solve the problems of blind retrieval, resource waste, rigid intent understanding and / or rough answer fusion existing in the existing standard multi-modal RAG technology.

[0007] In order to achieve the above purpose, the present application adopts the following technical scheme: In a first aspect, a multi-modal hybrid retrieval enhancement generation method is provided, comprising: obtaining an original query sentence of a user; performing multi-modal query intent analysis on the original query sentence to obtain query intent weights of each modality in the multi-modal, wherein the multi-modal includes text, image, video and / or structured data; for each modality, positively allocating corresponding required computing resources for retrieval and / or priority levels used for retrieval according to the corresponding query intent weight, and based on the corresponding allocation result, performing retrieval in the corresponding database using the original query sentence to obtain the corresponding retrieval result; The retrieval results of the respective modalities are subjected to multi-modal information fusion and sorting processing to obtain a multi-modal content list in which the retrieval results of the respective modalities are sorted according to query intent weight, query relevance, information quality and / or information redundancy and complementarity, specifically including: for each piece of information obtained in the retrieval results of the respective modalities, the original query sentence and the corresponding information are respectively converted into vectors by a cross-modal encoder, and according to the two vectors obtained by the conversion, a query relevance score of the corresponding information with respect to the original query sentence is calculated, and an information quality score of the corresponding information is also obtained through information quality analysis; for each pair of pieces of information obtained in the retrieval results of the respective modalities, the corresponding information redundancy and complementarity relationship is analyzed in the following manner: if one of the corresponding two pieces of information is text / structured data information and describes the content of the other piece of information, the sorting priority of the text / structured data information is reduced or the text / structured data information is marked as a supplementary description of the other piece of information, and the sorting priority of the other piece of information is increased, and the other piece of information is associated with the text / structured data information, wherein the other piece of information is image / video information; the query intent weight of the respective modalities, the query relevance score and the information quality score of the information obtained in the respective retrievals, and the information redundancy and complementarity relationship of the information obtained in the respective pairs of retrievals are input into a pre-trained learning sorting model, and a multi-modal content list sorted according to query intent weight, query relevance, information quality and information redundancy and complementarity of the retrieval results of the respective modalities is output; The multi-modal content list is integrated into a prompt word, and the prompt word is input into a large language model or a large multi-modal model to output a multi-modal answer corresponding to the original query sentence and containing text, image, video and / or structured data; The multi-modal answer is transmitted to a human-computer interaction interface and presented to the user.

[0008] Based on the above invention content, a new scheme for multi-modal mixed retrieval enhancement generation based on multi-modal query intent analysis and multi-modal information fusion and sorting processing is provided, that is, first, the original query sentence of the user is subjected to multi-modal query intent analysis to obtain the query intent weight of each modality, then for each modality, the corresponding required calculation resources and / or priority level for retrieval are allocated in positive correlation according to the corresponding query intent weight, and independent retrieval is performed based on the corresponding allocation result to obtain the corresponding retrieval result, then multi-modal information fusion and sorting processing is performed on all retrieval results to obtain a multi-modal content list sorted according to the query intent weight, query relevance, information quality and / or information redundancy and complementarity, and finally the list is integrated into a prompt word and imported into a large language model or a large multi-modal model to output a multi-modal answer, so that the leap from passive and rigid blind retrieval to active and flexible intelligent decision-making can be realized, the response efficiency, answer accuracy and user experience of the system are significantly improved, and resources and retrieval costs are saved, facilitating practical application and promotion.

[0009] In one possible design, the original query sentence is subjected to multi-modal query intent analysis to obtain the query intent weight of each modality in multi-modal, including: The original query sentence is preprocessed to obtain a new query sentence, wherein the preprocessing includes stop word removal processing and / or spelling error correction processing; The new query sentence is encoded into a semantic vector by applying a pre-trained language model; The query target entity is identified from the new query sentence by applying an entity recognition technology; The query keyword with intent indication is extracted from the new query sentence by applying a keyword extraction technology; The semantic vector, the query target entity and the query keyword are imported into a pre-trained multi-label classification model to output the query intent weight of each modality in multi-modal, wherein the multi-modal includes text, image, video and / or structured data.

[0010] In one possible design, for each modality, the original query sentence is used to perform retrieval in the corresponding database to obtain the corresponding retrieval result, including: Based on hybrid retrieval technology and combined with keyword matching and vector similarity search, the original query sentence is used to perform retrieval in a text database to obtain a text retrieval result; And / or, based on hybrid retrieval technology and combined with keyword matching and vector similarity search, the original query sentence is used to perform retrieval in a structured database to obtain a structured data retrieval result; and / or, transforming the original query sentence into a vector by a cross-modal encoder, and then searching for the most similar image vector to the vector in an image vector database to obtain an image search result; and / or, transforming the original query sentence into a vector by a cross-modal encoder, and then searching for the most similar video vector to the vector in a video vector database to obtain a video search result.

[0011] In one possible design, when the learning ranking model has the capability of making layout suggestions for the retrieved information according to the query intent weight, query relevance, information quality and / or information redundancy and complementarity, at least one of the retrieved information in the multi-modal content list is marked with a layout suggestion.

[0012] In one possible design, the multi-modal answer is transmitted to a human-computer interaction interface and presented to the user, including: transmitting the multi-modal answer to a human-computer interaction interface; presenting the multi-modal answer to the user in a rich text or card layout on the human-computer interaction interface.

[0013] In a second aspect, a multi-modal hybrid retrieval enhancement generation apparatus is provided, including a query sentence acquisition unit, a query intent analysis unit, a retrieval allocation execution unit, a fusion ranking processing unit, a multi-modal answer generation unit and an answer transmission and presentation unit; The query sentence acquisition unit is configured to acquire an original query sentence of a user. The query intent analysis unit is communicatively connected to the query sentence acquisition unit and configured to perform multi-modal query intent analysis on the original query sentence to obtain query intent weights of each modality in multi-modal, wherein the multi-modal includes text, image, video and / or structured data. The retrieval allocation execution unit is respectively communicatively connected to the query intent analysis unit and the query sentence acquisition unit, and configured to, for each modality, positively allocate corresponding required computing resources and / or priority levels for retrieval according to the corresponding query intent weight, and based on the corresponding allocation result, perform retrieval in a corresponding database using the original query sentence to obtain a corresponding retrieval result. The fusion sorting processing unit is in communication connection with the search distribution execution unit, and is configured to perform multi-modal information fusion and sorting processing on the search results of the respective modalities to obtain a multi-modal content list in which the search results of the respective modalities are sorted according to query intention weight, query relevance, information quality and / or information redundancy and complementarity, specifically including: for each search result in the search results of the respective modalities, converting the original query sentence and the corresponding information into vectors respectively by using a cross-modal encoder, and calculating a query relevance score of the corresponding information and the original query sentence according to the two vectors obtained by the conversion, and further obtaining an information quality score of the corresponding information by information quality analysis; for each pair of search results in the search results of the respective modalities, analyzing the information redundancy and complementarity relationship of the corresponding information in the following manner: if one of the corresponding two information is text / structured data information and describes the content of the other information, then the sorting priority of the text / structured data information is reduced or the text / structured data information is marked as a supplementary description of the other information, and the sorting priority of the other information is improved, and the other information is associated with the text / structured data information, wherein the other information is image / video information; introducing the query intention weight of the respective modalities, the query relevance score and the information quality score of the each search result, and the information redundancy and complementarity relationship of the each pair of search results into a pre-trained learning sorting model, and outputting to obtain a multi-modal content list in which the search results of the respective modalities are sorted according to query intention weight, query relevance, information quality and information redundancy and complementarity; The multi-modal answer generation unit is in communication connection with the fusion sorting processing unit, and is configured to integrate the multi-modal content list into a prompt word, and introduce the prompt word into a large language model or a large multi-modal model to output a multi-modal answer corresponding to the original query sentence and containing text, image, video and / or structured data. The answer transmission and presentation unit is in communication connection with the multi-modal answer generation unit, and is configured to transmit the multi-modal answer to a human-computer interaction interface and present the multi-modal answer to the user.

[0014] In a third aspect, the present application provides a computer device, comprising a storage module, a processing module and a transceiver module which are in communication connection in sequence, wherein the storage module is configured to store a computer program, the transceiver module is configured to transmit and receive messages, and the processing module is configured to read the computer program and execute the multi-modal hybrid search enhancement generation method according to any possible design of the first aspect.

[0015] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon instructions which, when executed on a computer, perform the multi-modal hybrid retrieval enhancement generation method according to the first aspect or any possible design of the first aspect.

[0016] In a fifth aspect, the present application provides a computer program product comprising a computer program or instructions which, when executed on a computer, implement the multi-modal hybrid retrieval enhancement generation method according to the first aspect or any possible design of the first aspect.

[0017] Advantages of the above-mentioned scheme are as follows: (1) The present application provides a new scheme of multi-modal hybrid retrieval enhancement generation based on multi-modal query intent analysis and multi-modal information fusion and sorting processing, that is, first performing multi-modal query intent analysis on the original query sentence of the user to obtain the query intent weight of each modality, then for each modality, positively allocating the corresponding required computing resources and / or priority level of retrieval according to the corresponding query intent weight, and performing independent retrieval based on the corresponding allocation result to obtain the corresponding retrieval result, then performing multi-modal information fusion and sorting processing on all retrieval results to obtain a multi-modal content list sorted according to the query intent weight, query relevance, information quality and / or information redundancy and complementarity, and finally integrating the list into a prompt word and importing it into a large language model or a large multi-modal model to output a multi-modal answer, so as to realize the leap from passive and rigid blind retrieval to active and flexible intelligent decision-making, significantly improve the response efficiency, answer accuracy and user experience of the system, and save resources and retrieval costs; (2) The effectiveness and intuitiveness of the answer can be greatly improved by matching the most suitable data modality for the query question, thereby improving the one-time solution rate of the question; (3) Unnecessary global retrieval on all modality databases can be avoided, reducing the consumption of computing resources and speeding up the response speed; (4) It can provide stereoscopic answers with pictures, texts and even videos, far beyond pure text customer service, like an experienced customer service expert; (5) The present scheme is not simply an "addition" of multi-modal retrieval, but introduces an "intelligent brain" - multi-modal query intent analysis, which can make the optimal retrieval decision before retrieval, solve the defects of "blind retrieval" and "rigid rules" in the prior art, and through multi-modal information fusion and sorting processing, the retrieved information can also be finely processed, solving the problem of chaotic information presentation and poor user experience in the prior art, facilitating practical application and promotion. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0019] Figure 1 The flowchart of the multi-modal mixed retrieval enhanced generation method provided by the embodiments of the present application.

[0020] Figure 2 The structural diagram of the multi-modal mixed retrieval enhanced generation device provided by the embodiments of the present application.

[0021] Figure 3 The structural diagram of the computer device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. It should be noted that the description of these embodiment modes is used to help understand the present application, but does not constitute a limitation on the present application.

[0023] It should be understood that although the terms first and second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object can be called the second object, and similarly, the second object can be called the first object, without departing from the scope of the example embodiments of the present application.

[0024] It should be understood that for the term "and / or" which may appear in the present application, it is only a description of the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can mean that A exists alone, B exists alone or A and B exist together, etc. For example, A, B and / or C can mean that any one of A, B and C exists or any combination thereof; for the term " / and" which may appear in the present application, it is another description of the relationship of another associated object, which means that there can be two kinds of relationships, for example, A / and B can mean that A exists alone or A and B exist together; in addition, for the character " / " which may appear in the present application, it generally means that the associated objects before and after are in an "or" relationship.

[0025] Embodiments As Figure 1 shown, the multi-modal hybrid retrieval enhanced generation method provided by the first aspect of the embodiments can be, but is not limited to, executed by a computer device with certain computing resources, such as a server, a personal computer (PC, referring to a multi-purpose computer with size, price and performance suitable for personal use), a smart phone, a personal digital assistant (PDA) or a wearable device, etc. Figure 1 As

[0026] S1. Obtain an original query sentence of a user.

[0027] In the step S1, the original query sentence can be, but is not limited to, obtained through a conventional man-machine interactive interface, for example, the user can input a text string "show me the latest A model unmanned aerial vehicle operating in the mountain area video and list its main performance parameters" through a keyboard as a complex and possibly implicit multi-modal demand original query sentence.

[0028] S2. Perform multi-modal query intention analysis on the original query sentence to obtain query intention weights of each modality in multi-modal, wherein the multi-modal includes but is not limited to text, image, video and / or structured data, etc.

[0029] In the step S2, the purpose of the multi-modal query intention analysis is not to directly generate retrieval instructions, but to "think" what the user really wants first, and output a weight vector containing different data modalities and their corresponding priority retrieval strategies (i.e. a vector composed of the query intention weights of each modality) in order to realize the conversion of the user's natural language query into retrieval instructions with different data modalities. Specifically, the multi-modal query intention analysis on the original query sentence to obtain the query intention weights of each modality in multi-modal includes but is not limited to the following steps S21-S25.

[0030] S21. Preprocess the original query sentence to obtain a new query sentence, wherein the preprocessing includes but is not limited to stop word removal processing and / or spelling error correction processing, etc.

[0031] In the step S21, the preprocessing is also a data cleaning process, wherein the specific process of the stop word removal processing and the spelling error correction processing, etc. are all prior art means, which will not be described here.

[0032] S22. Apply a pre-trained language model to encode the new query sentence into a semantic vector.

[0033] In the step S22, the language model is used to achieve deep semantic understanding of the new query sentence, so as to obtain the semantic vector with high latitude. In detail, the language model can be but not limited to the BERT (Bidirectional Encoder Representations from Transformers, which is a pre-trained language model based on the Transformer architecture proposed by Google in 2018, which can revolutionize the performance of natural language processing tasks through a bidirectional context understanding mechanism) model or the Ernie (Enhanced Representation Learning for Natural Language Inference) model, etc. obtained through conventional training.

[0034] S23. Apply entity recognition technology to identify query target entities from the new query sentence.

[0035] In the step S23, the entity recognition technology (Entity Recognition) is a key technology in natural language processing that locates and classifies named entities in text, mainly identifying predefined category entities such as names, places, and organizations. For example, based on the query sentence example in step S1, specific entities such as "A model drone" and "mountainous area" can be identified.

[0036] S24. Apply keyword extraction technology to extract query keywords with intent indication from the new query sentence.

[0037] In the step S24, the keyword extraction technology is a core task in the field of natural language processing, mainly extracting representative terms from text through statistical features, graph models, and semantic analysis techniques. For example, based on the query sentence example in step S1, keywords such as "look", "video", "list", and "parameter" can be extracted, where "look" / "video" can strongly point to the video modality, and "list" / "parameter" can strongly point to the text or structured data modality.

[0038] S25. Import the semantic vector, the query target entity, and the query keyword into a pre-trained multi-label classification model, and output the query intent weight in each modality in the multi-modal, wherein the multi-modal includes but is not limited to text, image, video, and / or structured data, etc.

[0039] In the step S25, the multi-label classification model can be but not limited to a Transformer-based classifier or a gradient boosting tree model, which is conventionally trained. The sample data required for the training can be in the form of: model input items containing semantic vectors, query target entities and query keywords, and model output items containing query intent weights of each modality and pre-labeled. The final output of the multi-label classification model is not a single decision, but a weighted modality list to represent the query intent weights of each modality. For example, the modality list is {video: 0.95, structured_data: 0.9, text: 0.8, image: 0.3}. This modality list (i.e. weight vector) can accurately guide the subsequent modality-based retrieval, i.e. it tells that the video database and structured database should be searched first, the text database second, and the image database last.

[0040] S3. For each modality, the corresponding required computing resources for retrieval and / or the priority level for retrieval are positively correlated with the corresponding query intent weight, and based on the corresponding allocation result, the original query sentence is used to perform retrieval in the corresponding database to obtain the corresponding retrieval result.

[0041] In the step S3, the specific allocation strategy is: the higher the query intention weight, the more computing resource allocation and / or the higher priority level configuration required for the corresponding modal retrieval; and the lower the query intention weight, the less computing resource allocation and / or the lower priority level configuration required for the corresponding modal retrieval; in this way, the differential retrieval tasks can be allocated to different modal databases for retrieval according to the weight vector, achieving the purposes of non-blind retrieval and resource saving. Since the retrieval tasks are specifically performed independently in each modal database, the entire step is a heterogeneous retrieval, that is, for each modal, the original query statement is used to perform retrieval in the corresponding database to obtain the corresponding retrieval result, including but not limited to: using the original query statement to perform retrieval in the text database based on the hybrid retrieval technology combined with keyword matching and vector similarity search to obtain the text retrieval result; and / or, using the original query statement to perform retrieval in the structured database based on the hybrid retrieval technology combined with keyword matching and vector similarity search to obtain the structured data retrieval result; and / or, converting the original query statement into a vector through a cross-modal encoder, and then searching for the most similar image vector in the image vector database to obtain the image search result; and / or, converting the original query statement into a vector through a cross-modal encoder, and then searching for the most similar video vector in the video vector database to obtain the video search result. The hybrid retrieval technology is a technology that combines the advantages of full-text retrieval and vector retrieval, can realize the complementarity of precise matching and semantic understanding, and aims to improve the accuracy and comprehensiveness of information retrieval; the keyword matching can but is not limited to use the existing BM25 (Best Matching 25) algorithm, which is used to calculate the relevance score between query (query) and document (doc); the vector similarity search can but is not limited to use the cosine similarity calculation formula to realize; the cross-modal encoder can but is not limited to use the encoder structure in the CLIP (Contrastive Language-Image Pre-training, a multi-modal neural network model developed by OpenAI, which can realize the cross-modal semantic association between image and text through contrastive learning, and supports image classification and retrieval under zero-shot learning) model to realize; the specific retrieval task execution process is a prior art means, and will not be described here. In addition, if the query intention weight of a certain modal is lower than a preset threshold (for example, 0.1), the retrieval of the modal can be skipped.

[0042] S4. Multi-modal information fusion and sorting processing is performed on the retrieval results of each modal to obtain a multi-modal content list sorted according to the query intention weight, query relevance, information quality and / or information redundancy and complementarity of the retrieval results of each modal.

[0043] In the step S4, the multi-modal information fusion and sorting processing is used to integrate (specifically including screening, deduplication and sorting of all information materials, and planning how to best combine them together) the scattered information materials (for example: 2 video links, 5 text paragraphs and 1 parameter table picture, etc.) retrieved from different databases into a high-quality draft answer. Specifically, the multi-modal information fusion and sorting processing is performed on the retrieval results of the various modalities to obtain a multi-modal content list in which the retrieval results of the various modalities are sorted according to the query intention weight, query relevance, information quality and / or information redundancy and complementarity, including but not limited to the following steps S41-S43.

[0044] S41. For each retrieval information in the retrieval results of the various modalities, the original query sentence and the corresponding information are respectively converted into vectors by a cross-modal encoder, and the query relevance score of the corresponding information with respect to the original query sentence is calculated according to the two converted vectors, and the information quality score of the corresponding information is also obtained through information quality analysis.

[0045] In the step S41, the cross-modal encoder can also but not limited to adopt the encoder structure in the CLIP model to realize; the query relevance score can also but not limited to be calculated by using the cosine similarity calculation formula; and the specific process of the information quality analysis is also a prior art means, for example, first analyzing to obtain the picture definition or video length, and then calculating the information quality score based on the analysis result and a preset formula (generally, the higher the picture definition or the longer the video length, the higher the information quality score; the lower the picture definition or the shorter the video length, the lower the information quality score). In addition, if the query relevance score of a certain retrieval information with respect to the original query sentence is lower than a preset score threshold, the certain retrieval information can also be filtered out.

[0046] S42. For each pair of retrieval information in the retrieval results of the various modalities, the corresponding information redundancy and complementarity relationship can also but not limited to be analyzed in the following manner: if one of the corresponding two information is text / structured data information and describes the content of the other information, the sorting priority of the text / structured data information is reduced or the text / structured data information is marked as a supplementary description of the other information, the sorting priority of the other information is improved, and the other information is associated with the text / structured data information, wherein the other information is image / video information.

[0047] S43. The query intent weight of each modality, the query relevance score and information quality score of each retrieved information, and the information redundancy and complementarity relationship of each pair of retrieved information are input into a pre-trained learning ranking model, and a multi-modal content list ranked according to the query intent weight, query relevance, information quality, and information redundancy and complementarity of the retrieved results of each modality is output.

[0048] In the step S43, the learning ranking (LTR) model is a ranking model automatically optimized by machine learning technology in the fields of information retrieval, recommendation system, and natural language processing, etc., and the core is to learn a ranking function through training data to sort a collection of unordered or partially ordered items (such as web pages, goods, and news, etc.) according to a specific target to improve user experience, so that the learning ranking model can be conventionally trained to have the ability to rank the retrieved results of each modality according to the query intent weight, query relevance, information quality, and information redundancy and complementarity. In order to provide layout suggestions to the subsequent answer generation step, preferably, when the learning ranking model also has the ability to provide layout suggestions for retrieved information according to the query intent weight, query relevance, information quality, and / or information redundancy and complementarity, at least one retrieved information in the multi-modal content list is marked with a layout suggestion, for example, for a pair of image and text, the suggestion is “insert this chart after this text”. In addition, any one or any combination of the query intent weight of each modality, the query relevance score and information quality score of each retrieved information, and the information redundancy and complementarity relationship of each pair of retrieved information can be input into another pre-trained learning ranking model, and another multi-modal content list ranked according to the query intent weight, query relevance, information quality, and / or information redundancy and complementarity of the retrieved results of each modality is output.

[0049] S5. The multi-modal content list is integrated into a prompt word, and the prompt word is input into a large language model or a large multi-modal model to output a multi-modal answer corresponding to the original query sentence and containing text, image, video, and / or structured data.

[0050] In the step S5, the prompt words can be exemplified as: “Please generate a fluent and coherent answer according to the following materials: [Video 1], [Text summary A], [Parameter chart B]...”, so that the large language model or the large multi-modal model “strings” the separated information materials according to these prompt words, and generates the final answer containing text description, embedded picture / video link, etc. (i.e. seamlessly organizes these information materials into a logically coherent and easy-to-understand multi-modal answer). In addition, the large language model or the large multi-modal model is the model used in the existing standard multi-modal RAG technology, and the specific details are not described here.

[0051] S6. Transmitting the multi-modal answer to the human-computer interaction interface and presenting it to the user.

[0052] In the step S6, specifically, transmitting the multi-modal answer to the human-computer interaction interface and presenting it to the user includes but is not limited to: transmitting the multi-modal answer to the human-computer interaction interface; presenting the multi-modal answer to the user in a rich text (a text format containing multimedia elements such as format, style, image, link, etc., with more rich layout and display functions compared with pure text) or card layout on the human-computer interaction interface.

[0053] Therefore, the multi-modal hybrid retrieval enhancement generation method based on the foregoing steps S1-S6 provides a new scheme for multi-modal hybrid retrieval enhancement generation based on multi-modal query intent analysis and multi-modal information fusion and sorting processing, that is, first performing multi-modal query intent analysis on the user's original query sentence to obtain the query intent weight of each modality, then for each modality, according to the corresponding query intent weight, positively relatedly allocating the corresponding required computing resources and / or priority level for retrieval, and based on the corresponding allocation result, performing independent retrieval to obtain the corresponding retrieval result, then performing multi-modal information fusion and sorting processing on all retrieval results to obtain a multi-modal content list sorted according to the query intent weight, query relevance, information quality and / or information redundancy and complementarity of the retrieval result, and finally integrating the list into prompt words and importing them into a large language model or a large multi-modal model to output a multi-modal answer, so that the passive and rigid blind retrieval can be upgraded to active and flexible intelligent decision-making, the response efficiency, answer accuracy and user experience of the system are significantly improved, and resources and retrieval costs are saved, facilitating actual application and promotion.

[0054] As Figure 2As shown, the second aspect of the present embodiment provides a virtual device for implementing the multi-modal hybrid retrieval enhanced generation method of the first aspect, comprising a query sentence acquisition unit, a query intention analysis unit, a retrieval allocation execution unit, a fusion sorting processing unit, a multi-modal answer generation unit, and an answer transmission and presentation unit; The query sentence acquisition unit is configured to acquire an original query sentence of a user; The query intention analysis unit is in communication connection with the query sentence acquisition unit and is configured to perform multi-modal query intention analysis on the original query sentence to obtain query intention weights of each modality in multi-modalities, wherein the multi-modalities include text, image, video, and / or structured data; The retrieval allocation execution unit is in communication connection with the query intention analysis unit and the query sentence acquisition unit respectively, and is configured to, for each modality, positively allocate corresponding required computing resources for retrieval and / or priority levels for retrieval according to the corresponding query intention weight, and based on the corresponding allocation result, perform retrieval in the corresponding database using the original query sentence to obtain a corresponding retrieval result; The fusion sorting processing unit is in communication connection with the retrieval allocation execution unit and is configured to perform multi-modal information fusion and sorting processing on the retrieval results of each modality to obtain a multi-modal content list in which the retrieval results of each modality are sorted according to query intention weight, query relevance, information quality, and / or information redundancy and complementarity, specifically including: for each retrieval result in the retrieval results of each modality, converting the original query sentence and the corresponding information into vectors respectively by a cross-modal encoder, and calculating a query relevance score of the corresponding information and the original query sentence according to the two vectors obtained by the conversion, and further obtaining an information quality score of the corresponding information through information quality analysis; for each pair of retrieval results in the retrieval results of each modality, the corresponding information redundancy and complementarity relationship is analyzed as follows: if one of the two corresponding information is text / structured data information and describes the content of the other information, the sorting priority of the text / structured data information is reduced or the text / structured data information is marked as a supplementary description of the other information, and the sorting priority of the other information is improved, and the other information is associated with the text / structured data information, wherein the other information is image / video information; the query intention weight of each modality, the query relevance score and the information quality score of each retrieval result, and the information redundancy and complementarity relationship of each pair of retrieval results are input into a pre-trained learning sorting model to output a multi-modal content list in which the retrieval results of each modality are sorted according to query intention weight, query relevance, information quality, and information redundancy and complementarity; The multi-modal answer generation unit, in communication connection with the fusion ranking processing unit, is configured to integrate the multi-modal content list into a prompt word, and to input the prompt word into a large language model or a large multi-modal model to output a multi-modal answer corresponding to the original query sentence and containing text, image, video and / or structured data. The answer delivery and presentation unit, in communication connection with the multi-modal answer generation unit, is configured to deliver the multi-modal answer to a human-computer interaction interface and present it to the user.

[0055] The working process, working details and technical effects of the foregoing device provided by the second aspect of the present embodiment can be referred to the multi-modal hybrid retrieval enhancement generation method described in the first aspect, and will not be repeated here.

[0056] As shown in Figure 3 The third aspect of the present embodiment provides a computer device for executing the multi-modal hybrid retrieval enhancement generation method described in the first aspect, which comprises a storage module, a processing module and a transceiver module in communication connection in sequence, wherein the storage module is configured to store a computer program, the transceiver module is configured to transceive messages, and the processing module is configured to read the computer program and execute the multi-modal hybrid retrieval enhancement generation method described in the first aspect. Specifically, the storage module can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a first-in first-out memory (FIFO) and / or a first-in last-out memory (FILO), etc.; and the processing module can adopt, but is not limited to, a microprocessor with model number STM32F105 series. In addition, the computer device can further include, but is not limited to, a power module, a display screen and other necessary components.

[0057] The working process, working details and technical effects of the foregoing computer device provided by the third aspect of the present embodiment can be referred to the multi-modal hybrid retrieval enhancement generation method described in the first aspect, and will not be repeated here.

[0058] The fourth aspect of the embodiment provides a computer readable storage medium storing instructions of the multi-modal hybrid search enhancement generation method as described in the first aspect, that is, the computer readable storage medium stores instructions, and when the instructions are executed on a computer, the multi-modal hybrid search enhancement generation method as described in the first aspect is executed. Wherein, the computer readable storage medium refers to a carrier storing data, which can include, but is not limited to, floppy disks, optical disks, hard disks, flash memories, USB flash disks and / or Memory Stick, etc. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0059] The working process, working details and technical effects of the aforementioned computer readable storage medium provided by the fourth aspect of the embodiment can be referred to the multi-modal hybrid search enhancement generation method as described in the first aspect, which will not be described here.

[0060] The fifth aspect of the embodiment provides a computer program product including a computer program or instructions, which, when executed by a computer, implement the multi-modal hybrid search enhancement generation method as described in the first aspect. Wherein, the computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0061] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal hybrid search enhancement generation method, characterized by, The method comprises the following steps: obtaining an original query sentence of a user; performing multi-modal query intent analysis on the original query sentence to obtain query intent weights of each modality in the multi-modal, wherein the multi-modal includes text, image, video and / or structured data; for each modality, positively assigning corresponding retrieval required computing resources and / or retrieval priority level according to the corresponding query intent weight, and based on the corresponding assignment result, performing retrieval in the corresponding database using the original query sentence to obtain corresponding retrieval results; performing multi-modal information fusion and sorting processing on the retrieval results of each modality to obtain a multi-modal content list sorted according to query intent weight, query relevance, information quality and / or information redundancy and complementarity of the retrieval results of each modality, specifically including: for each retrieval result in the retrieval results of each modality, converting the original query sentence and the corresponding information into vectors respectively through a cross-modal encoder, and calculating the query relevance score of the corresponding information and the original query sentence according to the two vectors obtained by conversion, and further obtaining the information quality score of the corresponding information through information quality analysis; for each pair of retrieval results in the retrieval results of each modality, the corresponding information redundancy and complementarity relationship is analyzed in the following way: if one of the corresponding two information is text / structured data information and describes the content of the other information, the sorting priority of the text / structured data information is reduced or the text / structured data information is marked as a supplementary description of the other information, and the sorting priority of the other information is improved, and the other information is associated with the text / structured data information, wherein the other information is image / video information; the query intent weight of each modality, the query relevance score and the information quality score of each retrieval result, and the information redundancy and complementarity relationship of each pair of retrieval results are input into a pre-trained learning sorting model, and a multi-modal content list sorted according to query intent weight, query relevance, information quality and information redundancy and complementarity of the retrieval results of each modality is output; integrating the multi-modal content list into a prompt word, and inputting the prompt word into a large language model or a large multi-modal model to output a multi-modal answer corresponding to the original query sentence and containing text, image, video and / or structured data; transmitting the multi-modal answer to a human-computer interaction interface and presenting it to the user.

2. The multimodal hybrid search boost generation method of claim 1, wherein, The multi-modal query intent analysis on the original query sentence to obtain the query intent weights of each modality in the multi-modal includes: preprocessing the original query sentence to obtain a new query sentence, wherein the preprocessing includes stop word removal processing and / or spelling error correction processing; encoding the new query sentence into a semantic vector by applying a pre-trained language model; identifying query target entities from the new query sentence by applying entity recognition technology; extracting a query keyword indicative of an intent from the new query sentence by using a keyword extraction technology; importing the semantic vector, the query target entity and the query keyword into a pre-trained multi-label classification model to output query intent weights in each modality in a multi-modal, wherein the multi-modal includes text, image, video and / or structured data.

3. The multimodal hybrid search boost generation method of claim 1, wherein, For each modality, the original query sentence is used to perform retrieval in the corresponding database to obtain the corresponding retrieval result, including: performing retrieval in a text database using the original query sentence based on a hybrid retrieval technology combined with keyword matching and vector similarity search to obtain a text retrieval result; and / or, performing retrieval in a structured database using the original query sentence based on a hybrid retrieval technology combined with keyword matching and vector similarity search to obtain a structured data retrieval result; and / or, converting the original query sentence into a vector through a cross-modal encoder, and then searching for the most similar image vector to the vector in an image vector database to obtain an image search result; and / or, converting the original query sentence into a vector through a cross-modal encoder, and then searching for the most similar video vector to the vector in a video vector database to obtain a video search result.

4. The multimodal hybrid search boost generation method of claim 1, wherein, When the learning ranking model has the ability to make layout suggestions for the retrieved information according to query intent weights, query relevance, information quality and / or information redundancy and complementarity, at least one retrieved information in the multi-modal content list is marked with a layout suggestion.

5. The multimodal hybrid search boost generation method of claim 1, wherein, The multi-modal answer is transmitted to a human-computer interaction interface and presented to the user, including: transmitting the multi-modal answer to a human-computer interaction interface; presenting the multi-modal answer to the user in a rich text or card layout on the human-computer interaction interface.

6. A multi-modal hybrid search enhancement generation apparatus, characterized by, including a query sentence acquisition unit, a query intent analysis unit, a retrieval allocation and execution unit, a fusion ranking processing unit, a multi-modal answer generation unit and an answer transmission and presentation unit; The query sentence acquisition unit is configured to acquire the original query sentence of the user. The query intent analysis unit is communicatively connected to the query sentence acquisition unit and is configured to perform multi-modal query intent analysis on the original query sentence to obtain query intent weights in each modality in a multi-modal, wherein the multi-modal includes text, image, video and / or structured data. The retrieval allocation and execution unit is respectively communicatively connected to the query intent analysis unit and the query sentence acquisition unit, and is configured to, for each modality, positively allocate corresponding retrieval required computing resources and / or retrieval priority levels according to the corresponding query intent weights, and based on the corresponding allocation results, use the original query sentence to perform retrieval in the corresponding database to obtain the corresponding retrieval result. The fusion sorting processing unit is in communication connection with the search distribution execution unit, and is configured to perform multi-modal information fusion and sorting processing on the search results of the respective modalities to obtain a multi-modal content list in which the search results of the respective modalities are sorted according to query intention weight, query relevance, information quality and / or information redundancy and complementarity. Specifically, for each search result in the search results of the respective modalities, the original query sentence and the corresponding information are respectively converted into vectors by a cross-modal encoder, and a query relevance score of the corresponding information and the original query sentence is calculated according to the two vectors obtained by the conversion. In addition, an information quality score of the corresponding information is obtained through information quality analysis. For each pair of search results in the search results of the respective modalities, the information redundancy and complementarity relationship of the corresponding information is analyzed in the following manner: if one of the corresponding two information is text / structured data information and describes the content of the other information, the sorting priority of the text / structured data information is reduced or the text / structured data information is marked as a supplementary description of the other information, and the sorting priority of the other information is improved, and the other information is associated with the text / structured data information, wherein the other information is image / video information. The query intention weight of the respective modalities, the query relevance score and the information quality score of each search result, and the information redundancy and complementarity relationship of each pair of search results are input into a pre-trained learning sorting model to output a multi-modal content list in which the search results of the respective modalities are sorted according to query intention weight, query relevance, information quality and information redundancy and complementarity. The multi-modal answer generation unit is in communication connection with the fusion sorting processing unit, and is configured to integrate the multi-modal content list into a prompt word, and input the prompt word into a large language model or a large multi-modal model to output a multi-modal answer corresponding to the original query sentence and containing text, image, video and / or structured data. The answer transmission and presentation unit is in communication connection with the multi-modal answer generation unit, and is configured to transmit the multi-modal answer to a human-computer interaction interface and present the multi-modal answer to the user.

7. A computer device, comprising: The device comprises a storage module, a processing module and a transceiver module which are sequentially in communication connection. The storage module is configured to store a computer program. The transceiver module is configured to transceive messages. The processing module is configured to read the computer program and execute the multi-modal hybrid search enhancement generation method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that The computer readable storage medium stores instructions, and when the instructions are executed on a computer, the multi-modal hybrid search enhancement generation method according to any one of claims 1-5 is executed.

9. A computer program product comprising computer programs or instructions, characterized in that, The computer program or the instructions realize the multi-modal hybrid search enhancement generation method according to any one of claims 1-5 when executed on a computer.

Images