Search method, computer program product and electronic device

Abstract

The invention provides a search method, a computer program product and electronic equipment, and relates to the technical field of artificial intelligence, the search method comprises the following steps: obtaining a multi-modal knowledge base, and converting multi-modal information in the multi-modal knowledge base into a knowledge base text vector; the search input is converted, and converted text information corresponding to the search input is determined; determining multiple paths of candidate text information based on the knowledge base text vector, the converted text information and the multi-modal representation of the search input, and determining recall information according to the multiple paths of candidate text information; and inputting the search input and the recall information into a target model to obtain a query result corresponding to the search input. The accuracy of the query result can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a search method, a computer program product, and an electronic device. Background Art

[0002] With the development of large model technology, retrieval enhancement technology is becoming more and more critical.

[0003] In the related art, the retrieval enhancement technology is generally based on text. The text-based RAG system mainly processes text information and has certain limitations. The knowledge base usually used by the text-based RAG system is also mainly text information, which may lead to low accuracy of the query results. Summary of the invention

[0004] The purpose of the present disclosure is to provide a search method, a computer program product, and an electronic device, thereby overcoming the problem of low accuracy of query results caused by limitations and defects of related technologies, at least to a certain extent.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, there is provided a search method, which is applied to search for multimodal information in which the search input is at least text information and image information, and includes: acquiring a multimodal knowledge base, and converting the multimodal information in the multimodal knowledge base into a knowledge base text vector; converting the search input, and determining the converted text information corresponding to the search input; determining multiple candidate text information based on the knowledge base text vector, the converted text information, and the multimodal representation of the search input, and determining recall information based on the multiple candidate text information; inputting the search input and the recall information into a target model, and generating a query result corresponding to the search input.

[0007] In an exemplary embodiment of the present disclosure, the converting of the multimodal information in the multimodal knowledge base into a knowledge base text vector includes: converting the image information in the multimodal information into reference text information, and keeping the text information in the multimodal information unchanged; converting the reference text information and the text information into text vectors to determine the knowledge base text vector.

[0008] In an exemplary embodiment of the present disclosure, converting the image information in the multimodal information into reference text information includes: inputting the image information in the multimodal information and context information of the image information into a multimodal model to obtain the reference text information.

[0009] In an exemplary embodiment of the present disclosure, the multiple candidate text information includes a first candidate text information and a second candidate text information; the multiple candidate text information is determined based on the knowledge base text vector, the converted text information and the multimodal representation of the search input, and the recall information is determined based on the multiple candidate text information, including: performing text matching based on the knowledge base text vector and the converted text information corresponding to the text information in the search input to determine the first candidate text information; performing text matching based on the knowledge base text vector and the multimodal representation of the search input to determine the second candidate text information; and fusing the first candidate text information and the second candidate text information to determine the recall information.

[0010] In an exemplary embodiment of the present disclosure, the text matching is performed based on the knowledge base text vector and the converted text information corresponding to the text information in the search input to determine the first path of candidate text information, including: performing text vectorization on the converted text information to determine the text vector; determining the similarity between the knowledge base text vector of multiple information in a multimodal knowledge base and the text vector, and determining the first path of candidate text information based on the similarity.

[0011] In an exemplary embodiment of the present disclosure, determining the similarity between the knowledge base text vectors of multiple information in the multimodal knowledge base and the text vector includes: inputting the knowledge base text vectors of multiple information in the multimodal knowledge base and the text vector into a multimodal similarity discriminator for classification, and determining the classification result; determining the similarity based on the classification result.

[0012] In an exemplary embodiment of the present disclosure, the text matching is performed based on the knowledge base text vector and the search input to determine the second path of candidate text information, including: extracting a multimodal representation of the search input and the input position of the image information and the text information in the search input; determining the distance between the knowledge base text vector of multiple information in the multimodal knowledge base and the multimodal representation, and determining the second path of candidate text information based on the distance.

[0013] In an exemplary embodiment of the present disclosure, the inputting the search input and the recall information into a target model to generate a query result corresponding to the search input includes: splicing the recall information with the search input to determine splicing information, and inputting the splicing information into the target model to generate a query result corresponding to the search input.

[0014] According to one aspect of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the computer program implements any one of the above-mentioned search methods.

[0015] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned search methods by executing the executable instructions.

[0016] In the technical solution provided by the embodiments of the present disclosure, on the one hand, recall information is generated by combining the knowledge base text vectors corresponding to multimodal information such as image information and text information and multimodal query questions, and then retrieval enhancement is performed based on the recall information to determine the retrieval results corresponding to the search input. Since the multimodal knowledge base and the information of different modes in the search input are taken into consideration, the processing dimension is increased, the comprehensiveness and accuracy of the recall information are improved, and the accuracy and reliability of the query results are improved. On the other hand, since the search input can be processed, the limitations of text-based retrieval enhancement in related technologies are avoided, the content in the knowledge base can be fully covered, the dimensions of the search input are fully considered, the comprehensiveness is increased, the richness of the retrieval results is enhanced, and the retrieval enhancement generation effect is improved.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 A schematic diagram showing a system architecture to which the search method of an embodiment of the present disclosure can be applied.

[0020] Figure 2 A flowchart of a search method in an embodiment of the present disclosure is schematically shown.

[0021] Figure 3 A schematic diagram schematically illustrates rewriting search input in an embodiment of the present disclosure.

[0022] Figure 4 The following is a schematic diagram showing a process of determining multiple candidate text information in an embodiment of the present disclosure.

[0023] Figure 5 The diagram schematically illustrates the generation of query results according to an embodiment of the present disclosure.

[0024] Figure 6The overall process diagram of retrieval enhancement generation based on multimodal query in an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] Text-based retrieval-augmented generation (RAG) systems have shown good performance in tasks such as question answering and dialogue generation in open domains. However, from the perspective of multimodal retrieval and multimodal knowledge enhancement, there are the following problems: Inability to support multimodal retrieval: Text-based RAG systems mainly process text information, and such systems often cannot handle non-text information such as images well. For example, if a user's question is about a picture, a text-based RAG system may not be able to give an accurate answer. In actual search and question-answering scenarios, user queries are often multimodal, including text, images and other information. Insufficient multimodal knowledge enhancement: The knowledge bases usually used by text-based RAG systems are also mainly text information, and knowledge containing non-text information such as images is often not well utilized. In actual knowledge bases, rich multimodal information is often contained. If the RAG system can utilize this multimodal information, it may generate more accurate and richer answers.

[0028] In order to solve the technical problems existing in the related art, some embodiments of the present disclosure provide a search method that can be applied to a multimodal question-and-answer scenario. The multimodal question-and-answer scenario can be a question-and-answer scenario in any field, such as a customer service question-and-answer scenario in an application platform, or a question-and-answer scenario on a search engine, etc.

[0029] Figure 1 A schematic diagram of a system architecture to which the search method and device of some embodiments of the present disclosure can be applied is shown.

[0030] In some embodiments, the system architecture 100 on which the multimodal query depends may include a terminal 110 and a server 120. The terminal 110 may include any type of terminal. The terminal 110 and the server 120 may transmit information via a network 130. The server 120 is used to process the search input uploaded by the terminal 110 to obtain a query result.

[0031] It should be noted that in the above search method, the server 120 can be any type of server or server cluster. The terminal 110 can be any type of device. The above search method can be completely executed by the server; or partially executed by the server and partially executed by the terminal, which is not specifically limited here.

[0032] Next, refer to Figure 2 As shown, each step in the search method in the embodiment of the present disclosure is described in detail.

[0033] In step S210, a multimodal knowledge base is obtained, and multimodal information in the multimodal knowledge base is converted into a knowledge base text vector.

[0034] In some embodiments, the multimodal knowledge base may be a knowledge base used to generate additional inputs in the retrieval enhancement generation process in the multimodal question-answering system. Retrieval enhancement is used to enable the model to not only rely on its internal knowledge when generating answers, but also utilize external information sources, such as databases or knowledge bases, to provide more accurate and richer answers.

[0035] Multimodal question answering systems are designed to process and answer questions involving multiple modalities. For example, a user may ask a question about the content of an image, such as "Who is the person in this picture?" or "What style of building is in this picture?". To answer these questions, the system needs to understand both the text query and the image content.

[0036] Multimodal question answering systems can leverage information from other modalities, such as images and audio, to improve the relevance and accuracy of search results. For example, if a user enters "red car" in a search engine, a multimodal question answering system can combine text and image information to find pictures of real red cars.

[0037] It is worth noting that the knowledge base enhanced by multimodal retrieval is also multimodal. This multimodal knowledge base can provide richer and more comprehensive information, thereby helping the system to answer questions more accurately or find relevant search results.

[0038] The multimodal knowledge base may store documents that can be used as additional inputs. The multimodal database may contain multimodal information, including text information and image information. In some embodiments, the text information and image information in the multimodal knowledge base may be converted into corresponding knowledge base text vectors, and the knowledge base text vector refers to the vector corresponding to each information in the knowledge base.

[0039] Since multimodal information contains information of different modes such as image information and text information, different processing methods can be used to process information of different modes. For example, in order to more effectively utilize the knowledge in the image information in the multimodal knowledge base, the image information in the multimodal information can be converted into reference text information, and the text information in the multimodal information remains unchanged. Furthermore, the reference text information and the unchanged text information can be converted into text vectors to determine the knowledge base text vector.

[0040] Among them, the image information and the context information of the image information in the multimodal information are input into the multimodal model to obtain reference text information. The multimodal model can be an open source multimodal large model (VL-LLM), for example, it can be a Qwen-VL model. When the multimodal model converts the image information into text, the image information and the context information of the image information can be input into the multimodal model to help the model better understand and extract information that is more relevant to the context information, thereby avoiding redundant information. Exemplarily, attention calculations can be performed on the image information and the context information based on the multimodal model to obtain reference text information corresponding to the image information. Then a complete text knowledge base is obtained based on the reference text information and the original text information.

[0041] Furthermore, for the reference text information and the text information, the reference text information and the text information can be converted into text vectors through a text vector model to obtain a knowledge base text vector, and the knowledge base vector is stored in the knowledge base. The text vector model can be, for example, a BGE model or other models as long as it can perform text vector conversion.

[0042] In step S220, the search input is converted to determine converted text information corresponding to the search input.

[0043] In the disclosed embodiment, the search input may be an input query question, and the search input may be multimodal information including image information and text information. In addition, the search input may also be voice query information, and when it is voice query information, the voice query information may be converted into text information. When in different application platforms, the search input may be input by different objects. For example, when on the merchant side of a certain application platform, the search input may be input by the merchant; when on the user side of a certain application platform, the search input may be input by the user.

[0044] When the search input is obtained, the search input can be converted to merge the image information and text information in the search input into a converted text information. The search input conversion here can be understood as search input rewriting, and the goal of search input rewriting is to merge the image information and text information in the search input into a sentence. The converted text information obtained after rewriting can merge the contents of the image information and the text information. For example, the converted text information obtained after rewriting can be a sentence, which can include image information and text information. Based on the converted text information obtained by rewriting, the user's query intention can be expressed only by relying on the converted text information. It should be noted that this refers to the overall converted text information corresponding to the search input.

[0045] Exemplarily, attention calculation can be performed on the image information and the text information based on the multimodal model, so as to understand and rewrite the image information and the text information based on the multimodal model to obtain the converted text information corresponding to the search input. The multimodal model can be a Qwen-VL model.

[0046] refer to Figure 3 As shown in , if a search input is received, the image information of the search input includes relevant information of the new customer exclusive / yoga / fitness / Pilates comprehensive small group class, and the text information of the search input is "This does not show the score, please check what's going on?" Based on this, the image information and the text information can be input into the multimodal model, and the converted text information can be output. The output converted text information can be a comprehensive text that integrates the image information and the text information, for example, "The new customer exclusive / yoga / fitness / Pilates comprehensive small group class does not show the score, please check what's going on?".

[0047] Next, continue to refer to Figure 2 As shown in, in step S230, multiple candidate text information is determined based on the knowledge base text vector, the converted text information and the multimodal representation of the search input, and recall information is determined based on the multiple candidate text information.

[0048] In the disclosed embodiment, since the search input may contain text information and image information, multiple candidate text information may be determined based on the text information and the multimodal representation of the search input. The multimodal representation of the search input may be used to represent the overall representation vector of the search input. The multiple candidate text information may, for example, include a first candidate text information and a second candidate text information. When determining the multiple candidate text information based on the local dimension represented by the text information and the overall dimension represented by the search input, the first candidate text information may be determined based on the text information in the search input, and the second candidate text information may be determined based on the multimodal representation of the search input. The multiple candidate text information may be information extracted from a multimodal knowledge base.

[0049] Figure 4 A flowchart for determining multiple candidate text information is schematically shown in FIG. Figure 4 As shown in , it mainly includes the following steps:

[0050] In step S410, text matching is performed based on the knowledge base text vector and the converted text information corresponding to the text information in the search input to determine the first path of candidate text information;

[0051] In step S420, text matching is performed based on the knowledge base text vector and the multimodal representation of the search input to determine the second path of candidate text information;

[0052] In step S430, the first path of candidate text information and the second path of candidate text information are merged to determine the recall information.

[0053] In the disclosed embodiment, first, a first path of candidate text information can be generated based on the text information in the search input. Exemplarily, the conversion text information corresponding to the text information can be text vectorized through a text vector model to determine the text vector. The conversion text information corresponding to the text information can be a part of the overall conversion text information obtained in step S210. Furthermore, the knowledge base text vectors of multiple information in the multimodal knowledge base can be text matched with the text vectors of the text information to determine the similarity between the knowledge base text vectors of multiple information in the multimodal knowledge base and the text vectors of the text information. The similarity here can be cosine similarity, and can also be other types of similarity. Here, the similarity is taken as cosine similarity as an example for explanation.

[0054] When determining the similarity between the knowledge base text vectors of multiple information in the multimodal knowledge base and the text vectors of the text information, the similarity calculation can be implemented based on the multimodal similarity discriminator. Exemplarily, the knowledge base text vectors of multiple information in the multimodal knowledge base and the text vectors corresponding to the text information can be input into the multimodal similarity discriminator for classification to determine the classification results, and the similarity is determined based on the classification results. Among them, the classification can be 0 / 1 classification. When the classification result is the first result, it can be considered that the two are similar. When the classification result is the second result, it can be considered that the two are not similar. For example, the first result can be 1 and the second result can be 0.

[0055] In some embodiments, the similarity can be calculated by a multimodal similarity discriminator. Exemplarily, a data set can be pre-annotated, and the historical multimodal information and the information in the knowledge base are paired and input into the model of the ViBERT architecture during training to perform 0 / 1 classification to obtain a reference classification result. If the information in the knowledge base and the historical multimodal information are similar, the reference classification result output is 1, and if they are not similar, the reference classification result output is 0. Furthermore, the model parameters of the model can be adjusted based on the reference classification result to obtain a multimodal similarity discriminator.

[0056] After the similarity is determined, the first path of candidate text information can be selected in descending order of similarity. The number of the first path of candidate text information can be 3-6, or can be set to other values ​​according to actual needs.

[0057] For the overall search input, text matching can be performed based on the knowledge base text vector and the search input to determine the second path of candidate text information. Exemplarily, for the overall search input, a multimodal representation is extracted at the input position of the image information and the text information in the search input; the distance between the knowledge base text vector and the multimodal representation of multiple information in the multimodal knowledge base is determined, and the second path of candidate text information is determined based on the distance. In some embodiments, for the overall multimodal information, the representations corresponding to the image information and the text information, that is, the vectors of the corresponding token positions, can be taken out at the corresponding input positions of the image information and the text information to determine the multimodal representation.

[0058] After determining the multimodal representation corresponding to the search input, the multimodal representation can be multimodally matched with the knowledge base text vectors of multiple information in the multimodal knowledge base to determine the distance between the two. Further, the second path of candidate text information can be determined based on the distance. Among them, the number of second-path candidate text information can be the same as the number of first-path candidate text information, or it can be different. The number of second-path candidate text information can be determined according to actual needs, for example, it can be 3-6. In addition, multiple optional quantities can be determined, and further based on the accuracy of the multiple optional quantities, the number of second-path candidate text information can be determined based on the accuracy, which is not specifically limited here.

[0059] By calculating the similarity between the knowledge base text vectors of multiple information in the multimodal knowledge base and the text vectors of the text information through text matching, a first path of candidate text information is obtained, and by calculating the distance between the knowledge base text vectors of multiple information in the multimodal knowledge base and the search input through multimodal matching, a second path of candidate text information is obtained, so that multiple paths of candidate text information can be accurately extracted from the multimodal knowledge base.

[0060] Since the multimodal search input containing text information and image information input by the user and the multimodal information of the knowledge base are processed, the search input and the image information in the knowledge base are maximized, the search scope is increased, and the comprehensiveness is improved. In addition, by realizing the first recall from the text information dimension and the second recall from the overall dimension composed of image information and text information, the multimodal two-way recall method provides richer relevant knowledge for large model generation, improves the recall effect, improves the accuracy of search results, and improves the search quality.

[0061] After determining the number of first-way candidate text information and the number of second-way candidate text information, the first-way candidate text information and the second-way candidate text information can be merged and deduplicated, and the merged and deduplicated first-way candidate text information and the merged and deduplicated second-way candidate text information can be fused to determine the recall information from the multimodal knowledge base. For example, if the first-way candidate text information contains the same information as the second-way text information, the same information in the first-way text information or the second-way text information can be deleted, and the same information in the other text information can be retained. Further, the merged and deduplicated first-way candidate text information and the merged and deduplicated second-way candidate text information can be spliced ​​to obtain the recall information. For example, the first-way candidate text information can be arranged after the second-way candidate text information; the second-way candidate text information can also be arranged after the first-way candidate text information, and the order of splicing the merged and deduplicated first-way candidate text information and the merged and deduplicated second-way candidate text information is not specifically limited here. By merging, removing duplication, and fusing the first path of candidate text information and the second path of candidate text information, recall information is determined from the multimodal knowledge base, thereby improving the accuracy of the recall information.

[0062] Continue to refer Figure 2 As shown in , in step S240, the search input and the recall information are input into a target model to generate a query result corresponding to the search input.

[0063] In the disclosed embodiments, the recall information determined from the multimodal knowledge base can be spliced ​​with the search input and input into the target model so as to perform retrieval enhancement generation through the target model, thereby obtaining the query results corresponding to the search input. The target model can be a language model, such as a ChatGPT model. Exemplarily, the recall information can be spliced ​​with one or more of the text information and image information in the search input and input into the target model. When the search input is spliced ​​as a whole, the recall information can also be spliced ​​with the converted text information of the search input, which is not specifically limited here.

[0064] Retrieval-Augmented Generation (RAG) is a technology that combines retrieval and generation. It is mainly used to solve open-domain question-answering and dialogue generation tasks. In some embodiments, retrieval-augmented generation mainly consists of two parts: a retrieval module and a generation model. In the retrieval-augmented generation model, the retrieval module is used to retrieve the documents most relevant to the current question or dialogue context from a large-scale document database. This retrieval module is usually based on information retrieval algorithms such as vectors or ES, or it can be a retrieval model based on a neural network. Furthermore, the retrieved documents will be fed into a generation module, which is usually a sequence-to-sequence (Seq2Seq) model, such as ChatGPT. The generation module will generate an answer or dialogue response based on these documents and the current question or dialogue context.

[0065] Based on this, in the embodiments of the present disclosure, information associated with the current search input can be retrieved from the multimodal knowledge base as recall information, the recall information can be spliced ​​with the search input, and the obtained spliced ​​information can be input into the target model for result generation. Exemplarily, the recall information can be spliced ​​with the search input input by the user to form a new enhanced prompt as a prompt word input into the target model, so as to guide the target model to generate query results for the search input according to the prompt word. The enhanced prompt word is input into the target model, and the target model generates the final query result according to the input recall information and the search input.

[0066] For example, refer to Figure 5 As shown in , the first candidate text information includes information 1, information 2 and information 3, and the second candidate text information includes information 4, information 5 and information 6. Information 1, information 2, information 3, information 4, information 5 and information 6 can be spliced ​​to obtain recalled information. Further, the recalled information can be spliced ​​with the search input, and the spliced ​​information obtained by splicing is input into the target model to generate a query result corresponding to the multi-modal query information.

[0067] In the disclosed embodiments, the multimodal question answering system is designed to process and answer questions involving multiple modalities. For example, the search input may be a question about the content of an image, such as "Who is the person in this picture?" or "What is the style of the building in this picture?" In order to answer these questions, the system needs to understand both the text query and the image content.

[0068] Multimodal retrieval systems can use information from other modalities such as images and audio to improve the relevance and accuracy of search results. For example, if a user enters "red car" in a search engine, a multimodal retrieval system can combine text and image information to find pictures of real red cars.

[0069] At the same time, the knowledge information contained in the multimodal knowledge base may also contain information of multiple modalities. For example, a search input may contain a person's name (text information), a photo (image information) or some other tables introducing knowledge. Based on this, the knowledge base text vector of the multimodal knowledge base can be obtained, and the converted text information corresponding to the search input can be obtained, and matching is performed based on the converted text information and the knowledge base text vector. Based on the text information and image information in the search input, the first candidate text information and the second candidate text information are determined from multiple dimensions by relying on text and multimodal information, and then the first candidate text information and the second candidate text information are fused to obtain the recall information, and the corresponding query results are obtained based on the recall information and the original search input.

[0070] Figure 6 The overall process diagram of retrieval enhancement generation based on multimodal query is schematically shown in Figure 6 As shown in , for the multimodal information in the multimodal knowledge base, the image information of the multimodal information in the multimodal knowledge base can be converted into multimodal information text based on the VL-LLM model to obtain reference text information, and keep the original text information in the multimodal information unchanged. The reference text information and the text information are further vectorized to generate knowledge base text vectors of multiple information in the multimodal knowledge base, such as V1, V2, and V3.

[0071] The search input may include text information and image information. For example, the text information may be “This does not display the score. Please check it out. What’s going on?” The image information may be a product display list of a certain application platform. For the search input, the VL-LLM model may also be used to rewrite the search input in multimodal knowledge to achieve multimodal information textualization and obtain converted text information. The converted text information obtained by rewriting the multimodal information may be “New Customer Exclusive / Yoga / Fitness / Pilates Comprehensive Small Group Class does not display the score. Please check it out. What’s going on?” The converted text information corresponding to the text information is further vectorized to generate a text representation, that is, a text vector corresponding to the text information. At the same time, for the overall search input, the search input may be vectorized, and the corresponding multimodal representation may be extracted at the input position of the image information and the text information in the search input.

[0072] After obtaining the knowledge base text vector of the multimodal knowledge base, the text vector of the text information, and the multimodal representation of the search input, the first path of candidate text information can be determined based on the text information based on the knowledge base text vector and the similarity between the text vectors. Based on the distance between the knowledge base text vector and the multimodal representation, multimodal matching is performed based on the search input to determine the second path of candidate text information.

[0073] Based on this, the first candidate text information and the second candidate text information can be spliced ​​together to determine the recall information from the multimodal knowledge base. Furthermore, the recall information and the original search input can be spliced ​​together and input into the target model LLM to generate the query result corresponding to the search input. The query result can be, for example, "Hello, this type of store is not participating in the event. Currently, there are XXX types of stores participating in the event."

[0074] In the disclosed embodiment, the user's search input and the multimodal information of the multimodal knowledge base are processed to maximize the use of image information, and by designing a multimodal two-way recall method to obtain multi-way candidate text information, the recall information is determined based on the multi-way recall, and the knowledge recall effect and recall quality are effectively improved. Due to the effective use of the multimodal knowledge base and the image information in the search input, the content of the recall information is enriched and the comprehensiveness of the recall judgment is improved. In addition, a multimodal two-way recall system is introduced, and multimodal rewriting and recall technology is used to enhance the recall capability, and provide richer relevant knowledge for large model generation for retrieval enhancement, which improves the comprehensiveness of the query results and the quality of the query results, and enhances the richness of the query results.

[0075] In some embodiments of the present disclosure, a search device is provided, which may include: a knowledge base conversion module, an information conversion module, a recall information determination module, and an information query module, wherein:

[0076] A knowledge base conversion module, used to obtain a multimodal knowledge base and convert the multimodal information in the multimodal knowledge base into a knowledge base text vector;

[0077] An information conversion module, used to convert the search input and determine the converted text information corresponding to the search input;

[0078] A recall information determination module, configured to determine multiple candidate text information based on the knowledge base text vector, the converted text information and the multimodal representation of the search input, and determine recall information based on the multiple candidate text information;

[0079] The information query module is used to input the search input and the recall information into the target model to generate a query result corresponding to the search input.

[0080] In an exemplary embodiment of the present disclosure, the knowledge base conversion module includes: an image conversion module, which is used to convert the image information in the multimodal information into reference text information and keep the text information in the multimodal information unchanged; a text vector conversion module, which is used to convert the reference text information and the text information into text vectors to determine the knowledge base text vector.

[0081] In an exemplary embodiment of the present disclosure, the image conversion module includes: a conversion control module, which is used to input the image information in the multimodal information and the context information of the image information into the multimodal model to obtain the reference text information.

[0082] In an exemplary embodiment of the present disclosure, the multiple candidate text information includes a first candidate text information and a second candidate text information; the recall information determination module includes: a first determination module, used to perform text matching based on the knowledge base text vector and the converted text information corresponding to the text information in the search input, to determine the first candidate text information; a second determination module, used to perform text matching based on the knowledge base text vector and the multimodal representation of the search input, to determine the second candidate text information; a fusion module, used to fuse the first candidate text information and the second candidate text information to determine the recall information.

[0083] In an exemplary embodiment of the present disclosure, the first determination module includes: a text vectorization module, used to perform text vectorization on the converted text information and determine the text vector; a similarity determination module, used to determine the similarity between the knowledge base text vector of multiple information in the multimodal knowledge base and the text vector, and determine the first path of candidate text information based on the similarity.

[0084] In an exemplary embodiment of the present disclosure, the similarity determination module includes: a classification module, which is used to input the knowledge base text vectors of multiple information in the multimodal knowledge base and the text vector into the multimodal similarity discriminator for classification to determine the classification result; a similarity configuration module, which is used to determine the cosine similarity between the knowledge base text vector and the text vector according to the classification result.

[0085] In an exemplary embodiment of the present disclosure, the second determination module includes: a multimodal representation extraction module, used to extract a multimodal representation of the search input at the input position of the image information and text information contained in the search input; a distance calculation module, used to determine the distance between the knowledge base text vector of multiple information in the multimodal knowledge base and the multimodal representation, and determine the second-path candidate text information based on the distance.

[0086] In an exemplary embodiment of the present disclosure, the information query module is configured to: splice the recall information with the search input to determine splicing information, and input the splicing information into the target model to generate a query result corresponding to the search input.

[0087] It should be noted that the specific details of each part of the above-mentioned search device have been described in detail in some implementation methods of the corresponding methods. The undisclosed details can be found in the implementation content of the method part, and will not be repeated here.

[0088] The exemplary embodiment of the present disclosure also provides an electronic device. The electronic device may be the above-mentioned terminal device or server. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned search method by executing the executable instructions. In addition, the electronic device may also include a display for displaying an operation interface.

[0089] Below, an electronic device is exemplarily described in the form of a general-purpose computing device. The electronic device is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0090] Components of the electronic device may include, but are not limited to: the at least one processing unit mentioned above, the at least one storage unit mentioned above, a bus connecting different system components (including the storage unit and the processing unit), and a display unit.

[0091] The storage unit stores a program code, which can be executed by the processing unit, so that the processing unit performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit can perform the following steps: Figure 2 Follow the steps shown in .

[0092] The storage unit may include a readable medium in the form of a volatile memory unit, such as a random access memory unit (RAM) and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0093] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0094] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0095] The electronic device can also communicate with one or more external devices (such as keyboards, pointing devices, Bluetooth devices, etc.), and can also communicate with one or more devices that enable users to interact with the electronic device, and / or communicate with any device (such as routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. This communication can be carried out through an input / output (I / O) interface. In addition, the electronic device can also communicate with one or more networks (such as local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter. As shown in the figure, the network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0096] It should be noted that, in some embodiments of the present disclosure, a computer program product is also provided. The computer program product includes a computer program. When the computer program is executed by a processor, the above method is implemented.

[0097] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0098] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0099] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0100] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure.

[0101] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0102] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0103] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A search method, characterized in that: Multimodal information applied to search input that is at least text information and image information, including: Acquire a multimodal knowledge base, and convert the multimodal information in the multimodal knowledge base into a knowledge base text vector; Converting the search input to determine converted text information corresponding to the search input; Determine multiple candidate text information based on the knowledge base text vector, the converted text information and the multimodal representation of the search input, and determine recall information based on the multiple candidate text information; The search input and the recall information are input into a target model to generate a query result corresponding to the search input.

2. The search method according to claim 1, characterized in that: The step of converting the multimodal information in the multimodal knowledge base into a knowledge base text vector comprises: Converting the image information in the multimodal information into reference text information, and keeping the text information in the multimodal information unchanged; The reference text information and the text information are converted into text vectors to determine the knowledge base text vector.

3. The search method according to claim 2, characterized in that: The converting the image information in the multimodal information into reference text information comprises: The image information in the multimodal information and the context information of the image information are input into the multimodal model to obtain the reference text information.

4. The search method according to claim 1, characterized in that: The multiple candidate text information includes a first candidate text information and a second candidate text information; The determining of multiple candidate text information based on the knowledge base text vector, the converted text information and the multimodal representation of the search input, and determining the recall information according to the multiple candidate text information includes: Perform text matching based on the knowledge base text vector and the converted text information corresponding to the text information in the search input to determine the first path of candidate text information; Performing text matching based on the knowledge base text vector and the multimodal representation of the search input to determine a second path of candidate text information; The first candidate text information and the second candidate text information are merged to determine the recall information.

5. The search method according to claim 4, characterized in that: The step of performing text matching according to the knowledge base text vector and the converted text information corresponding to the text information in the search input to determine the first path of candidate text information includes: Performing text vectorization on the converted text information to determine a text vector; Determine the similarity between the knowledge base text vectors of multiple information in the multimodal knowledge base and the text vector, and determine the first path of candidate text information according to the similarity.

6. The search method according to claim 5, characterized in that: The determining of the similarity between the knowledge base text vectors of the plurality of information in the multimodal knowledge base and the text vector comprises: Inputting the knowledge base text vectors of the multiple information in the multimodal knowledge base and the text vector into a multimodal similarity discriminator for classification, and determining the classification result; The cosine similarity between the knowledge base text vector and the text vector is determined according to the classification result.

7. The search method according to claim 4, characterized in that: The performing text matching according to the knowledge base text vector and the multimodal representation of the search input to determine the second path of candidate text information includes: For the search input, extracting a multimodal representation at input positions of image information and text information contained in the search input; Determine the distance between the knowledge base text vector of the plurality of information in the multimodal knowledge base and the multimodal representation, and determine the second path of candidate text information according to the distance.

8. The search method according to claim 1, characterized in that: The step of inputting the search input and the recall information into a target model to generate a query result corresponding to the search input includes: The recalled information is concatenated with the search input to determine concatenated information, and the concatenated information is input into the target model to generate a query result corresponding to the search input.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the search method according to any one of claims 1 to 8 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the search method according to any one of claims 1 to 8 by executing the executable instructions.

Images