A multimodal information retrieval method and system based on large language model

Abstract

The present invention relates to the technical field of natural language processing, and specifically to a multimodal information retrieval method and system based on a large language model, comprising the following steps: introducing the ability of a large language model to analyze the search intention of a user and outputting unified key search information; processing multimodal information through a multimodal embedding model to generate a unified high-dimensional vector and storing it in a vector database; and realizing efficient and accurate retrieval of text-multimodal data by combining structured retrieval with vector retrieval. The beneficial effects are as follows: the multimodal information retrieval method and system based on a large language model proposed in the present invention use a large model to fully mine and analyze the search intention of the user, use an embedding model to fully utilize cross-modal semantic association information, optimize the characteristics of fuzzy matching of vector retrieval, and innovatively propose a method combining structured search with vector retrieval, which greatly improves the accuracy of retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a multimodal information retrieval method and system based on a large language model. Background Art

[0002] Multimodal data covers various types of information, such as text, images, audio, video, etc. These data are diverse and colorful. However, traditional modality-specific information retrieval methods have gradually revealed their shortcomings in cross-modal correlation mining.

[0003] In the past, information retrieval mainly focused on single-modal data, such as text-based retrieval. However, information in the real world often exists in multiple forms, such as images, video comments, audio clips, etc. on social media. The rise of this multimodal data has limited single-modal retrieval methods in fully exploring the connections between information, resulting in a decrease in information retrieval accuracy and efficiency.

[0004] At the same time, multimodal data is characterized by the potential existence of semantic associations between different modalities, which may express similar meanings even in different forms. For example, the content in an image may be associated with a text description or an audio comment. However, traditional retrieval methods find it difficult to capture and utilize such cross-modal semantic associations, resulting in the one-sidedness and limitations of information retrieval. The increasing amount and variety of multimodal data further exacerbates the limitations of traditional retrieval methods. In large-scale multimodal datasets, traditional modality-specific retrieval methods become difficult to handle the complex relationships between data, thus affecting the quality and efficiency of information retrieval. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal information retrieval method and system based on a large language model to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a multimodal information retrieval method based on a large language model, the method comprising the following steps:

[0007] The ability of introducing a large language model analyzes the user's search intent and outputs unified search key information;

[0008] Process multimodal information through a multimodal embedding model to generate a unified high-dimensional vector and store it in a vector database;

[0009] By combining structured retrieval and vector retrieval, efficient and accurate retrieval of text-multimodal data can be achieved.

[0010] Preferably, the multimodal data is preprocessed by calling different tools and plug-ins and combining the data characteristics to generate metadata for each data.

[0011] Preferably, the embedding model is used to vectorize the image, video, audio, and short text data, and the intelligent agent and large model are used to extract the summary or questions of the long text data and vectorize them, and the obtained high-dimensional vectors and their metadata are packaged and stored in the vector database.

[0012] Preferably, the input information module is used to extract information from the search requirements input by the user, obtain the search content and metadata, and use the metadata to screen the data in the vector database.

[0013] Preferably, the search content is vectorized by a data search module to obtain its vector, and the vector is subjected to similarity search on the filtered data to obtain the final search result.

[0014] A multimodal information retrieval system based on a large language model, the system comprising a multimodal data processing module, an input information extraction module and a data retrieval module;

[0015] Multimodal data processing module, used to process data of different modes and vectorize them into vector database;

[0016] Input information extraction module, used to analyze user input content to extract its search content and metadata;

[0017] Data retrieval module, used to output the final retrieval results based on the retrieval content and metadata

[0018] Preferably, the multimodal data processing module vectorizes the input multimodal information through the ImageBind multimodal embedding model.

[0019] Preferably, the input information module obtains user input and obtains structured output through the GPT4 large model combined with the prompt word engineering, where the output includes two parts: search content and metadata.

[0020] Preferably, the data retrieval module searches for multimodal information in the database by combining structured retrieval with vector retrieval. First, the data is filtered by structured retrieval combined with metadata obtained by the input information module. Then, the search content obtained by the input information module is vectorized and similarity retrieval is performed in the filtered database.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The multimodal information retrieval method and system based on a large language model proposed in the present invention uses a large model to fully mine and analyze the user's search intention, uses an embedding model to fully utilize cross-modal semantic association information, optimizes the characteristics of fuzzy matching in vector retrieval, and innovatively proposes a method combining structured search with vector retrieval, which greatly improves the accuracy of retrieval. The multimodal data processing module effectively processes long text data, effectively improving the retrieval accuracy of such data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Embodiment 1

[0026] See also Figure 1 The present invention provides a technical solution: a multimodal information retrieval method and system based on a large language model, aiming to improve the accuracy of multimodal information retrieval. The present invention comprises three modules: a multimodal data processing module, an input information extraction module, and a data retrieval module.

[0027] The multimodal data processing module vectorizes the input multimodal information through multimodal embedding models such as ImageBind. Words, pictures, videos, audio and other data can be directly input into the embedding model to obtain vectors of uniform dimensions. The vectors are then packaged with their metadata (including but not limited to time, source, location, text and other information) and stored in vector databases such as Milvus. There are two processing methods for long text data, which can be selected according to specific business:

[0028] (1) Long texts can be segmented and then, through the intelligent agent (Agent) using prompt word engineering, the segments (chunks) combined with specific prompt templates are input into the large model one by one, and finally a summary of the entire article is output. The summary content is then vectorized, and the text content is stored in metadata and in a vector database.

[0029] (2) Long texts can be segmented and then, through the intelligent agent (Agent) using prompt word engineering, the segments (chunks) combined with specific prompt templates can be input into the large model one by one, multiple questions can be generated according to the content of the article, and then the questions can be vectorized, and the text content can be stored in metadata and in the vector database.

[0030] The input information module obtains user input and obtains structured output through large models such as GPT4 combined with prompt word engineering. The output includes two parts: search content (core content after analysis and processing by large models) and metadata (keyword information appearing in user input, such as time, source, location, etc.).

[0031] The data retrieval module searches for multimodal information in the database by combining structured search with vector search. First, the data is filtered by combining structured search with metadata obtained by the input information module. Then, the search content obtained by the input information module is vectorized and similarity search is performed in the filtered database. Finally, the most similar k multimodal information is obtained.

[0032] The specific steps of the present invention are as follows:

[0033] 1) Preprocess multimodal data by calling different tools and plug-ins and combining data characteristics to generate metadata for each data.

[0034] 2) Use the embedding model to vectorize data such as pictures, videos, audio, and short texts;

[0035] 3) Use intelligent agents and large models to extract summaries or questions from long text data and perform vector processing;

[0036] 4) Packing the high-dimensional vectors obtained in steps 2) and 3) and their metadata into a vector database;

[0037] 5) Using the input information module to extract information from the user's input search requirements to obtain search content and metadata;

[0038] 6) Use metadata to filter data in the vector database;

[0039] 7) vectorizing the search content through the data retrieval module to obtain its vector;

[0040] 8) Performing similarity search on the vector in 7) on the filtered data to obtain the final search result;

[0041] The device of the present invention comprises: a multimodal data processing module, an input information extraction module, and a data retrieval module. The multimodal data processing module is used to process data of different modes and vectorize and store them in a vector database, the input information extraction module is used to analyze user input content to extract its retrieval content and metadata, and the data retrieval module is used to output the final retrieval result according to the retrieval content and metadata.

[0042] The advantages of the present invention are that it uses a large model to fully mine and analyze the user's search intention, uses an embedding model to fully utilize cross-modal semantic association information, optimizes the characteristics of fuzzy matching in vector retrieval, and innovatively proposes a method that combines structured search with vector retrieval, which greatly improves the accuracy of retrieval. The multimodal data processing module effectively processes long text data, effectively improving the retrieval accuracy of such data.

[0043] Embodiment 2

[0044] On the basis of the first embodiment, in order to facilitate the understanding of the present invention, an example is given below, taking the e-commerce field search as an example:

[0045] 1) Pre-process product images, text and other data, and generate images and metadata (e.g., product price, origin, color, link, etc.) for each product by writing an output processing script.

[0046] 2) Use the ImageBind model to vectorize product images and other data;

[0047] 3) Pack the high-dimensional vector obtained in step 2) and its metadata and store them in the Milvus vector database;

[0048] 4) Use the input information module to extract information from the user's input search requirements and obtain search content and metadata - assuming the input is "I want a pair of blue sneakers with a circular logo within 1,000 yuan", the output search content: a pair of sneakers with a circular logo, metadata: {price range: 0-1,000, color: blue};

[0049] 5) Use metadata to filter the data in the vector database—the filtering condition is {price range: 0-1000, color: blue};

[0050] 6) Vectorize the search content through the data retrieval module to obtain its vector - vectorize the text "a pair of sports shoes with a circular logo";

[0051] 7) Perform similarity search on the vector in 7) on the filtered data to obtain the final search result - output the images, links, descriptions and other attributes of the k most similar products.

[0052] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal information retrieval method based on a large language model, characterized by: The method comprises the following steps: (1) Preprocess multimodal data by calling different tools and plug-ins and combining data characteristics to generate metadata for each data; use embedding models to vectorize images, videos, audio, and short text data; use intelligent agents and large models to extract summaries or questions from long text data and vectorize them; and package the obtained high-dimensional vectors and their metadata into a vector database; (2) Introduce the ability of a large language model to analyze the user's search intent and output unified search key information. Use the input information module to extract information based on the user's input search requirements to obtain search content and metadata. (3) Use metadata to filter the data in the vector database, vectorize the search content through the data retrieval module to obtain its vector, and perform similarity search on the vector after filtering to obtain the final search result.

2. A retrieval system for the multimodal information retrieval method based on a large language model according to claim 1, characterized in that: The system consists of a multimodal data processing module, an input information extraction module and a data retrieval module; Multimodal data processing module, used to process data of different modes and vectorize them into vector database; Input information extraction module, used to analyze user input content to extract its search content and metadata; Data retrieval module, used to output final retrieval results based on retrieval content and metadata; The multimodal data processing module vectorizes the input multimodal information through the ImageBind multimodal embedding model; The input information module obtains user input and obtains structured output through the GPT4 large model combined with the prompt word engineering. The output includes two parts: search content and metadata. The data retrieval module searches for multimodal information in the database by combining structured retrieval with vector retrieval. First, the data is filtered by structured retrieval combined with the metadata obtained by the input information module. Then, the search content obtained by the input information module is vectorized and similarity retrieval is performed in the filtered database.

Images