Multi-modal data retrieval method and device, storage medium and electronic equipment

Abstract

The invention discloses a multi-modal data retrieval method and device, a storage medium and electronic equipment, the method is applied to the field of data retrieval, and the method comprises the following steps: converting multi-modal data input by a user into a target feature vector through a multi-modal large model; calculating the similarity between the target feature vector and the vector to be retrieved based on a cosine similarity formula, and determining the number of retrieval results according to the similarity; under the condition that the number of the retrieval results is greater than a retrieval number threshold value, retrieving the target feature vector in a center vector library based on a reverse index algorithm to obtain a first retrieval list; and calculating the confidence corresponding to each retrieval result in the first retrieval list according to a preset retrieval threshold, and filtering the retrieval results with the confidence lower than the preset confidence threshold to obtain a target retrieval list. Through the method and the device, the problem of relatively low retrieval efficiency caused by huge data volume when image data or text data is retrieved in related technologies is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data retrieval, in particular, to a multi-modal data retrieval method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the rapid development of artificial intelligence, the informatization and intelligentization process of various industries is constantly advancing, and the spread of data has long been limited to text carriers. More and more carrier forms such as images, sounds, and videos have emerged. Massive multi-modal data is stored in different business systems of various industries. Traditional single-modal methods can only handle the retrieval and labeling of single-modal data, and it is difficult to meet the needs of multi-modal data retrieval. Therefore, the related industries have successively carried out research on cross-modal retrieval and joint analysis technology of multi-modal data. The data generated by the business systems in various enterprises with the development of informatization and digitization are increasingly rich and complex, including text, images, videos, sensor data and other modalities, which brings challenges to data processing and analysis. Traditional data management and analysis mode has been difficult to meet the needs of modern data services.

[0003] In related technologies, based on sample data of actual business scenarios, the scenes and target objects involved are complex and diverse, and the data sources are wide, the data information dimensions are not unified, and the data base is large. Therefore, there is a problem and pain point of slow retrieval speed in data retrieval.

[0004] In view of the problem of low retrieval efficiency caused by large amount of data when retrieving image data or text data in related technologies, no effective solution has been proposed so far. SUMMARY

[0005] The main purpose of the present application is to provide a multi-modal data retrieval method, device, storage medium and electronic device to solve the problem of low retrieval efficiency caused by large amount of data when retrieving image data or text data in related technologies.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a multi-modal data retrieval method is provided, which comprises: converting multi-modal data input by a user into a target feature vector by a multi-modal large model, wherein the multi-modal data comprises at least one of the following: text data, image data; calculating the similarity between the target feature vector and a to-be-retrieved vector based on a cosine similarity formula, and determining the number of retrieval results according to the similarity, wherein the to-be-retrieved vector is a feature vector obtained by performing feature extraction on data in a to-be-retrieved database using the multi-modal large model; in the case that the number of retrieval results is greater than a retrieval number threshold, retrieving the target feature vector in a center vector library based on an inverted index algorithm to obtain a first retrieval list, wherein the first retrieval list at least comprises: a plurality of retrieval results, and a confidence degree corresponding to each retrieval result; calculating the confidence degree corresponding to each retrieval result in the first retrieval list according to a preset retrieval threshold, filtering retrieval results with a confidence degree lower than a preset confidence threshold, and obtaining a target retrieval list.

[0007] Further, converting multi-modal data input by a user into a target feature vector by a multi-modal large model comprises: receiving multi-modal data input by a user, and generating a retrieval sentence according to the multi-modal data; wherein the multi-modal data further comprises: retrieval information, the retrieval information comprising at least one of the following: data source, data generation time, the preset retrieval threshold, and data type; inputting the multi-modal data into the multi-modal large model, extracting a first feature vector of the image data by an image encoder in the multi-modal large model, and extracting a second feature vector of the text data by a text encoder in the multi-modal large model; extracting a third feature vector of the retrieval sentence by the text encoder in the multi-modal large model; storing the first feature vector, the second feature vector, and the third feature vector to a preset storage space to obtain the target feature vector.

[0008] Further, retrieving the target feature vector in a center vector library based on an inverted index algorithm to obtain a first retrieval list comprises: classifying center vectors in the center vector library to obtain a plurality of clusters, and creating index information of each cluster in the plurality of clusters; determining a target cluster corresponding to the target feature vector according to the index information of each cluster; retrieving feature vectors similar to the target feature vector in the target cluster to obtain the first retrieval list.

[0009] Further, before retrieving the target feature vector in the center vector library based on the inverted index algorithm, the method further comprises: determining the number of center points according to the data volume of the database to be searched; performing dimension reduction processing and compression processing on the feature vectors in the database to be searched through a priority queue algorithm to obtain processed feature vectors to be searched; filtering center point vectors from the processed feature vectors to be searched based on the number of center points, and constructing a graph structure according to the center point vectors; and determining the center vector library according to the graph structure.

[0010] Further, filtering center point vectors from the processed feature vectors to be searched based on the number of center points and constructing a graph structure according to the center point vectors comprises: filtering center point vectors from the processed feature vectors to be searched based on the number of center points through a data filtering algorithm, wherein the data filtering algorithm comprises at least one of a clustering algorithm and a random sampling algorithm; constructing nodes corresponding to the center point vectors and calculating the similarity between the center point vectors; constructing edges between the nodes according to the similarity between the center point vectors; and constructing the graph structure according to the nodes corresponding to the center point vectors and the edges between the nodes.

[0011] Further, after determining the number of retrieval results according to the similarity, the method further comprises: in the case where the number of retrieval results is less than or equal to the retrieval quantity threshold, retrieving the target feature vector in the database to be searched to obtain a second retrieval list; sorting the retrieval results in the second retrieval list according to the cosine similarity to obtain a sorted second retrieval list; calculating the confidence of each retrieval result in the sorted second retrieval list according to a preset retrieval threshold, filtering retrieval results with a confidence lower than the preset confidence threshold, and obtaining the target retrieval list.

[0012] Further, calculating the confidence of each retrieval result in the first retrieval list according to a preset retrieval threshold, filtering retrieval results with a confidence lower than the preset confidence threshold, and obtaining a target retrieval list comprises: performing normalization processing on the confidence of each retrieval result in the first retrieval list according to the preset retrieval threshold to obtain the processed confidence of each retrieval result; determining retrieval results with a confidence higher than the preset confidence threshold in the first retrieval list, and obtaining the target retrieval list.

[0013] To achieve the above object, according to another aspect of the present application, a multi-modal data retrieval device is provided, which comprises: a conversion unit configured to convert multi-modal data input by a user into a target feature vector by a multi-modal large model, wherein the multi-modal data comprises at least one of the following: text data, image data; a calculation unit configured to calculate a similarity between the target feature vector and a to-be-retrieved vector based on a cosine similarity formula, and determine a number of retrieval results according to the similarity, wherein the to-be-retrieved vector is a feature vector obtained by performing feature extraction on data in a to-be-retrieved database by using the multi-modal large model; a first retrieval unit configured to, in a case where the number of retrieval results is greater than a retrieval number threshold, perform retrieval on the target feature vector in a center vector library based on an inverted index algorithm to obtain a first retrieval list, wherein the first retrieval list at least comprises: a plurality of retrieval results, and a confidence degree corresponding to each retrieval result; and a first filtering unit configured to calculate the confidence degree corresponding to each retrieval result in the first retrieval list according to a preset retrieval threshold, filter retrieval results with a confidence degree lower than a preset confidence threshold, and obtain a target retrieval list.

[0014] Further, the conversion unit comprises: a generation subunit configured to receive multi-modal data input by a user, and generate a retrieval sentence according to the multi-modal data; wherein the multi-modal data further comprises: retrieval information, the retrieval information comprising at least one of the following: data source, data generation time, the preset retrieval threshold, and data type; a first extraction subunit configured to input the multi-modal data into the multi-modal large model, extract a first feature vector of the image data by an image encoder in the multi-modal large model, and extract a second feature vector of the text data by a text encoder in the multi-modal large model; a second extraction subunit configured to extract a third feature vector of the retrieval sentence by the text encoder in the multi-modal large model; and a storage subunit configured to store the first feature vector, the second feature vector, and the third feature vector to a preset storage space to obtain the target feature vector.

[0015] Further, the first retrieval unit comprises: a classification subunit configured to classify center vectors in the center vector library to obtain a plurality of clusters, and create index information of each cluster in the plurality of clusters; a first determination subunit configured to determine a target cluster corresponding to the target feature vector according to the index information of each cluster; and a retrieval subunit configured to retrieve feature vectors similar to the target feature vector in the target cluster to obtain the first retrieval list.

[0016] Further, the apparatus further comprises: a first determination unit, configured to determine a number of center points according to a data volume of the database to be searched before searching the target feature vector in the center vector library based on an inverted index algorithm to obtain a first search list; a dimension reduction unit, configured to perform dimension reduction processing and compression processing on the feature vectors in the database to be searched by a priority queue algorithm to obtain processed feature vectors to be searched; a construction unit, configured to filter center point vectors from the processed feature vectors to be searched based on the number of center points, and construct a graph structure according to the center point vectors; and a second determination unit, configured to determine the center vector library according to the graph structure.

[0017] Further, the construction unit comprises: a filtering subunit, configured to filter center point vectors from the processed feature vectors to be searched based on the number of center points by a data filtering algorithm, wherein the data filtering algorithm comprises at least one of a clustering algorithm and a random sampling algorithm; a calculation subunit, configured to construct nodes corresponding to the center point vectors, and calculate similarities between the center point vectors; a first construction subunit, configured to construct edges between the nodes according to the similarities between the center point vectors; and a second construction subunit, configured to construct the graph structure according to the nodes corresponding to the center point vectors and the edges between the nodes.

[0018] Further, the apparatus further comprises: a second search unit, configured to search the target feature vector in the database to be searched to obtain a second search list after determining a number of search results according to the similarities, in a case where the number of search results is less than or equal to the search quantity threshold; an ordering unit, configured to order the search results in the second search list according to cosine similarity to obtain an ordered second search list; and a second filtering unit, configured to calculate a confidence degree corresponding to each search result in the ordered second search list according to a preset search threshold, filter search results with a confidence degree lower than the preset confidence threshold, and obtain the target search list.

[0019] Further, the first filtering unit comprises: a processing subunit, configured to perform normalization processing on a confidence degree corresponding to each search result in the first search list according to the preset search threshold to obtain a processed confidence degree of each search result; and a second determination subunit, configured to determine search results with a confidence degree higher than a preset confidence threshold in the first search list to obtain the target search list.

[0020] In order to achieve the above object, according to an aspect of the present application, a computer program product is provided, comprising a computer program which, when executed by a processor, implements any one of the above-mentioned retrieval methods of multi-modal data. The computer program, when executed by the processor, implements the steps of the retrieval method of multi-modal data described in various embodiments of the present application.

[0021] In order to achieve the above object, according to an aspect of the present application, a computer readable storage medium is provided, comprising stored computer instructions, wherein the computer instructions, when executed by a processor, implement any one of the above-mentioned retrieval methods of multi-modal data.

[0022] In order to achieve the above object, according to an aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, the memory being configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement any one of the above-mentioned retrieval methods of multi-modal data.

[0023] By the present application, the following steps are adopted: converting multi-modal data input by a user into a target feature vector by a multi-modal large model, wherein the multi-modal data comprises at least one of the following: text data, image data; calculating a similarity between the target feature vector and a to-be-retrieved vector based on a cosine similarity formula, and determining a number of retrieval results according to the similarity, wherein the to-be-retrieved vector is a feature vector obtained by feature extraction on data in a to-be-retrieved database using the multi-modal large model; in the case that the number of retrieval results is greater than a retrieval number threshold, retrieving the target feature vector in a central vector library based on an inverted index algorithm to obtain a first retrieval list, wherein the first retrieval list at least comprises: a plurality of retrieval results, and a confidence degree corresponding to each retrieval result; calculating the confidence degree corresponding to each retrieval result in the first retrieval list according to a preset retrieval threshold, filtering retrieval results with a confidence degree lower than a preset confidence threshold to obtain a target retrieval list, thereby solving the problem of low retrieval efficiency caused by large data volume when retrieving image data or text data in related technologies.

[0024] By converting multimodal information such as text data and image data input by users into target feature vectors through a multimodal large model, the data representation can be unified, so that different types of search requests can be efficiently processed under the same framework, achieving the technical effect of cross-modal information fusion retrieval. At the same time, by calculating the cosine similarity between the target feature vector and the vector to be retrieved in the database, the similarity between the data can be quickly evaluated. Furthermore, when the number of search results exceeds the preset threshold, a center point indexing algorithm is used in the center vector library for deep search. The first search list obtained not only contains multiple search results, but also adds the confidence of each result, which can greatly reduce the search scope and improve the search response speed and user experience. In addition, by calculating and filtering the confidence of each search result, eliminating entries with confidence below the threshold, and generating the final target search list, the high accuracy and reliability of the search results are ensured, high-quality data retrieval is achieved, and the technical effect of improving search efficiency and performance is further achieved. At the same time, it also reduces the cost of invalid data processing and promotes the efficient use of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0026] Figure 1 is a flowchart of a multimodal data retrieval method provided in accordance with the first embodiment of the present application;

[0027] Figure 2 1 is a schematic diagram of an optional process of screening a center point vector according to the first embodiment of the present application;

[0028] Figure 3 This is a schematic diagram of an optional process of selecting a center point vector from a feature vector to be retrieved after dimensionality reduction processing according to the first embodiment of the present application;

[0029] Figure 4 1 is a flow chart of an optional multimodal data retrieval method provided in accordance with the first embodiment of the present application;

[0030] Figure 5 is a schematic diagram of a multimodal data retrieval device provided according to the second embodiment of the present application;

[0031] Figure 6 This is a schematic diagram of an electronic device for retrieving multimodal data provided in accordance with the fifth embodiment of the present application. DETAILED DESCRIPTION

[0032] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0033] It should be noted that the user information (including but not limited to user equipment information, user personal information, collected data, used data, generated data, processed data, etc.) and data (including but not limited to data for analysis, stored data, displayed data, collected information, used information, generated information, processed information, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary security measures, do not violate public order and good customs, and provide corresponding operation portal for the user to choose authorization or refusal. For example, an interface is provided between the system and the relevant users or institutions, and before obtaining the relevant information, the interface needs to send an acquisition request to the aforementioned user or institution, and after receiving the consent information fed back by the aforementioned user or institution, the relevant information is acquired.

[0034] It should be noted that the present application provides a corresponding operation portal for the user to choose to agree or refuse the automatic decision result; if the user chooses to refuse, the expert decision process is entered.

[0035] In order to enable the persons skilled in the art to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the persons skilled in the art without creative labor should belong to the scope of protection of the present application.

[0036] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0037] Embodiment one

[0038] The application will be described in the following with reference to the preferred embodiments, Figure 1 is a flowchart of a retrieval method of multi-modal data according to Embodiment One of the present application, as shown in the figure, the method comprises the following steps: Figure 1

[0039] Step S101, converting the multi-modal data input by the user into a target feature vector by a multi-modal large model, wherein the multi-modal data comprises at least one of the following: text data, image data.

[0040] In this embodiment one, in order to realize intelligent retrieval of cross-modal data, it is necessary to use a multi-modal large model to uniformly convert different types of data input by the user (such as text, image) into a computer processable feature vector form. The multi-modal large model is used to process multiple data types, extract the semantic information of text and the visual features of image, and convert them into a set of numerical representations, i.e. feature vectors, so that the computing device can understand and match information of different modalities by calculating the distance or similarity between vectors. Illustratively, the user can input a text description or an image, and the multi-modal large model can convert it into a feature vector for finding the most similar matching item in a database containing text, image and even video, greatly enhancing the flexibility and efficiency of data retrieval.

[0041] Step S102, calculating the similarity between the target feature vector and the to-be-retrieved vector based on the cosine similarity formula, and determining the number of retrieval results according to the similarity, wherein the to-be-retrieved vector is a feature vector obtained by feature extraction of data in the to-be-retrieved database using the multi-modal large model.

[0042] In this embodiment one, the cosine similarity formula can be used to quantitatively compare the similarity between the target feature vector converted from the user input data and the feature vector extracted from each to-be-retrieved data in the to-be-retrieved database by the multi-modal large model. The cosine similarity reflects the consistency of their directions by calculating the cosine value of the included angle between two vectors, so as to evaluate the similarity between data. According to the calculated similarity value, the number of retrieval results that meet a certain threshold value is determined, i.e. the number of database vectors that have a similarity to the target vector reaching or exceeding a preset standard, so as to realize the function of accurately screening out relevant items from a large amount of data. This method can effectively process multi-modal data and improve the retrieval efficiency and accuracy.

[0043] Step S103, in the case where the number of retrieval results is greater than the retrieval number threshold, retrieving the target feature vector in the center vector library based on the inverted index algorithm to obtain a first retrieval list, wherein the first retrieval list at least includes: a plurality of retrieval results, and a confidence degree corresponding to each retrieval result.

[0044] ​In the first embodiment, when the number of retrieved results exceeds the preset retrieval threshold, further optimization of the search process in the central vector library is required. The central vector library is a database constructed in advance based on the central vectors in the database to be retrieved, wherein the central vectors are generated by processing part of the data to be retrieved in the database to be retrieved. The target feature vector is searched in the central vector library, and the cosine similarity is used as the basis to quickly locate similar central vectors, thereby obtaining the first retrieval list most relevant to the target feature vector. The list not only contains multiple search results, but also contains the confidence of each search result, that is, the evaluation of the matching degree of each search result with the user's search condition, ensuring the accuracy and reliability of the search. Through the above steps, the search efficiency can be significantly improved while maintaining the high quality of the search results when searching in a large amount of data.

[0045] In step S104, the confidence of each search result in the first retrieval list is calculated according to the preset search threshold, and the search results with a confidence lower than the preset confidence threshold are filtered to obtain the target retrieval list.

[0046] In the first embodiment, after obtaining the first retrieval list, the confidence of each search result in the first retrieval list needs to be evaluated and screened according to the preset search threshold, wherein the preset search threshold can be a search threshold input by the user or determined according to historical search conditions or expert experience. The confidence represents the matching degree between the search result and the user's search condition. By calculating the confidence of each search result and comparing it with the confidence threshold, search results with a confidence lower than the threshold can be filtered out, and only the most relevant and reliable items are retained, thereby obtaining the target retrieval list. Through the above steps, the accuracy of the search results can be ensured, the interference of low-quality data is avoided, and the user search experience and data utilization efficiency are improved.

[0047] Exemplarily, each search result in the target retrieval list and the confidence corresponding to each search result can be represented as: {“_score”: 0.28442588, “embedding”: “AKC4vgDgij0AQOC+......+”}. Wherein, “_score” represents the confidence, and “embedding” represents the feature vector corresponding to the search result.

[0048] In summary, the multi-modal data retrieval method provided by the embodiment one of the application, by the multi-modal large model, converts the multi-modal data input by the user into a target feature vector, wherein the multi-modal data includes at least one of the following: text data, image data; based on the cosine similarity formula, the similarity between the target feature vector and the to-be-retrieved vector is calculated, and the number of retrieval results is determined according to the similarity, wherein the to-be-retrieved vector is a feature vector obtained by performing feature extraction on the data in the to-be-retrieved database using the multi-modal large model; in the case that the number of retrieval results is greater than the retrieval number threshold, the target feature vector is retrieved in the center vector library based on the inverted index algorithm, and a first retrieval list is obtained, wherein the first retrieval list at least includes: a plurality of retrieval results, and a confidence degree corresponding to each retrieval result; according to the preset retrieval threshold, the confidence degree corresponding to each retrieval result in the first retrieval list is calculated, and the retrieval result with a confidence degree lower than the preset confidence threshold is filtered, and a target retrieval list is obtained, solving the problem that in the related art, when image data or text data is retrieved, due to the large amount of data, the retrieval efficiency is low.

[0049] By converting the multi-modal information such as text data and image data input by the user into a target feature vector through a multi-modal large model, the data representation form can be unified, so that different types of retrieval requests can be efficiently processed under the same framework, and the technical effect of cross-modal information fusion retrieval is achieved. At the same time, by calculating the cosine similarity between the target feature vector and the to-be-retrieved vector in the database, the similarity between the data can be quickly evaluated, and further, when the number of retrieval results exceeds the preset threshold, the center point index algorithm is used for deep retrieval in the center vector library, and the first retrieval list obtained not only contains a plurality of retrieval results, but also adds the confidence degree of each result, which can greatly reduce the retrieval range and improve the response speed of retrieval and user experience. In addition, by calculating and filtering the confidence degree of each retrieval result, the item with a confidence degree lower than the threshold is removed, and the final target retrieval list is generated, ensuring the high accuracy and reliability of the retrieval result, realizing high-quality data retrieval, further achieving the technical effect of improving the retrieval efficiency and performance, and also reducing the cost of processing invalid data, promoting the efficient use of data.

[0050] Optionally, in the multimodal data retrieval method provided in Example 1 of the present application, the multimodal data input by the user is converted into a target feature vector through a multimodal large model, including: receiving the multimodal data input by the user, and generating a search statement based on the multimodal data; wherein the multimodal data also includes: search information, the search information includes at least one of the following: data source, data generation time, preset search threshold, data type; inputting the multimodal data into the multimodal large model, extracting the first feature vector of the image data through the image encoder in the multimodal large model, extracting the second feature vector of the text data through the text encoder in the multimodal large model; extracting the third feature vector of the search statement through the text encoder in the multimodal large model; storing the first feature vector, the second feature vector and the third feature vector in a preset storage space to obtain the target feature vector.

[0051] In the first embodiment of the present invention, the user can input multimodal data including text descriptions and picture examples, and provide search information, such as data source, data generation time, preset search threshold and data type parameters. This information together constitutes a search statement, which sets clear conditions and scopes for subsequent search operations. For example, a search condition can be expressed as: {"dataSourceValue": 1, "importTime": "20250206000000, 20250212235959", "searchEmb": "Cars running on the road", "threshold": "0.1", "dataType": 0}. Among them, "dataSourceValue" represents the data source, "importTime" represents the data generation time range, "threshold" represents the above-mentioned preset search threshold, "searchEmb" represents text information, and "dataType" represents the data type.

[0052] Then, the multimodal large model is used to process the multimodal data input by the user. The multimodal large model can understand and process various data types (such as images and text) at the same time. It contains two key components: the image encoder and the text encoder. Among them, the image encoder is responsible for parsing and extracting the key features of the image data, converting it into the first feature vector, which is used to represent the main information of the image; the text encoder is used to analyze the text data and convert its semantic content into the second feature vector. Similarly, the text encoder is also used to process the search statement and convert it into the third feature vector to ensure that the search conditions and the data in the database to be searched can be compared in the same vector space.

[0053] Finally, the extracted first feature vector, second feature vector and third feature vector are integrated and stored in a preset storage space to form a target feature vector set. It should be noted that the feature vectors corresponding to the to-be-retrieved data in the to-be-retrieved database are also stored in the preset storage space for retrieval and calculation.

[0054] By generating a retrieval statement corresponding to the retrieval condition, the target feature vector can be generated, compared with the feature vectors stored in the to-be-retrieved database, and based on the cosine similarity calculation rule, the result most matched with the user's retrieval condition is quickly located and returned. This process realizes the conversion from the user's diversified input to the unified vector representation in the business system, not only improves the accuracy of retrieval, but also ensures the efficiency and flexibility of cross-modal data retrieval, and provides a more intelligent, fast and accurate retrieval experience for the user.

[0055] Optionally, in the multi-modal data retrieval method provided in Embodiment One of the present application, the target feature vector is retrieved in the center vector library based on the inverted index algorithm to obtain a first retrieval list, including: classifying the center vectors in the center vector library to obtain a plurality of clusters, and creating index information of each cluster in the plurality of clusters; determining a target cluster corresponding to the target feature vector according to the index information of each cluster; retrieving a feature vector similar to the target feature vector in the target cluster to obtain the first retrieval list.

[0056] In Embodiment One, in order to optimize the retrieval performance of large-scale databases, the inverted index algorithm can be used for data processing. First, the center vectors in the center vector library are classified and processed, and similar vectors are grouped together to form a plurality of clusters. Exemplarily, the plurality of clusters and the index information of each cluster can be as shown in Table 1. Each cluster is equivalent to a set of center vectors, which have high similarity in the vector space and can simplify the subsequent retrieval process. In order to speed up the retrieval, index information can be created for each cluster, so that the specific cluster can be quickly located according to the index information, thereby realizing efficient access to the vectors in the cluster.

[0057] Table 1

[0058] Index information Intra-cluster vector Intra-cluster vector Intra-cluster vector C_1 Id:code1 Id:code2 C_2 Id:code1 Id:code2 Id:code3 C_3 Id:code1 Id:code2 ... ... ... ... C_k Id:code1 Id:code2 C_1024 Id:code1 Id:code2 Id:code3

[0059] Then, the cosine similarity or Euclidean distance and other measurement methods can be used to calculate the similarity between the target feature vector provided by the user and the index information of each cluster to determine the target cluster most relevant to the target feature vector. This process effectively reduces the search range and avoids full-range search in a large database, significantly improving the retrieval speed.

[0060] Finally, fine search is performed in the determined target cluster to find the feature vector most similar to the target feature vector, and a first search list is constructed. This process usually involves calculating the similarity of the target feature vector with all the center vectors in the target cluster, then sorting them from high to low according to the similarity, and selecting the most relevant search results. The first search list not only contains the search results, but also can be accompanied by the confidence or similarity score of each result, providing data basis for subsequent search result filtering.

[0061] Through the above steps, the search range can be reduced, the massive data search demand can be processed in an efficient and accurate manner, and the technical effects of improving search efficiency and search result quality are achieved.

[0062] Optionally, in the multi-modal data search method provided in Embodiment One of the present application, before the target feature vector is searched in the center vector library based on the inverted index algorithm to obtain the first search list, the above method further includes: determining the number of center points according to the data amount of the database to be searched; performing dimension reduction processing and compression processing on the feature vectors in the database to be searched through a priority queue algorithm to obtain processed feature vectors to be searched; filtering center point vectors from the processed feature vectors to be searched based on the number of center points, and constructing a graph structure according to the center point vectors; and determining the center vector library according to the graph structure.

[0063] In Embodiment One, in order to effectively cope with the massive data search scene, a series of optimization measures can be taken to construct the center vector library, greatly improving the search speed and resource utilization efficiency. First, according to the actual data amount of the database to be searched, the number of center points is flexibly determined to ensure that the optimal balance point between search accuracy and calculation efficiency is found. It should be noted that the setting of the number of center points needs to consider both the quality of the search results and the running speed of the search algorithm. Too large may increase the amount of calculation, while too small may affect the search accuracy.

[0064] Then, the priority queue algorithm (Priority Queue, which can be abbreviated as PQ) is used to perform dimension reduction and compression processing on the feature vectors corresponding to the data in the database to be searched, so as to reduce the dimension and storage space of the vectors, and further improve the efficiency of the search algorithm. The priority queue algorithm can sort the feature vectors according to their importance and relevance, and select the most representative part to be retained, while the rest is compressed or ignored. This method not only guarantees the core information of the vectors, but also effectively reduces the burden of calculation and storage.

[0065] Secondly, based on the dimension-reduced and compressed feature vectors, the center point vectors are selected based on the preset number of center points. The graph structure is the core component of the center vector library, which takes each center point vector as a node and clearly describes the association relationship between the nodes to form a graph reflecting the association between the data. This graph structure can quickly locate the most relevant data area to the user query, greatly shortening the search path and time.

[0066] Finally, the set of center point vectors based on the graph structure is determined as the center vector library, providing an efficient and accurate data index for subsequent intelligent retrieval. By constructing the center vector library, not only the slow retrieval speed of massive data is solved, but also the accuracy of the retrieval results is optimized through the center points and the graph structure, providing a retrieval platform that can quickly respond and guarantee the quality of the results for users.

[0067] Optionally, in the multi-modal data retrieval method provided in Embodiment One of the present application, the center point vectors are selected from the processed feature vectors to be retrieved based on the number of center points, and the graph structure is constructed according to the center point vectors, including: selecting the center point vectors from the processed feature vectors to be retrieved based on the number of center points by a data selection algorithm, wherein the data selection algorithm includes at least one of the following: a clustering algorithm, a random sampling algorithm; constructing nodes corresponding to the center point vectors and calculating the similarity between the center point vectors; constructing edges between the nodes according to the similarity between the center point vectors; and constructing the graph structure according to the nodes corresponding to the center point vectors and the edges between the nodes.

[0068] In Embodiment One, when constructing the center vector library, the clustering algorithm or the random sampling algorithm can be used to select the center point vectors from the processed feature vectors to be retrieved. The clustering algorithm can group data according to the similarity or distance between vectors, while the random sampling algorithm selects the center point vectors by random selection to ensure the universality and representativeness of the samples.

[0069] Exemplarily, Figure 2 is an exemplary flowchart of selecting center point vectors according to Embodiment One of the present application. First, the dimensionality reduction processing is performed on the high-dimensional index vector data by the priority queue algorithm. Specifically, the N x 768-dimensional feature vectors are divided into N x 24 segments, each containing 32 dimensions. This step helps to reduce the data dimension and improve the efficiency of subsequent processing. The clustering algorithm is performed on the 32-dimensional feature vectors of the N x 24 sub-segments to group similar vectors into a class. 256 class centers are generated by clustering, which represent the mean or typical features of the vectors in their respective classes and are helpful for subsequent construction of the center vector library.

[0070] Further, Figure 3is a flowchart of an optional process of screening center point vectors from the dimension-reduced feature vectors to be searched according to Embodiment One of the present application. Figure 3 The numerous black dots scattered on the bottommost rectangular plane are data points corresponding to the dimension-reduced feature vectors to be searched in the optimized queue algorithm in the database to be searched. The bottommost feature vectors to be searched start to undergo the screening of the clustering algorithm, i.e., N times of iterative clustering, where N can be an integer greater than 1. After each clustering, representative intermediate data points are extracted from the numerous data points in the current level, which are then moved to the rectangular plane in the upper level. Then, the intermediate data points screened in the previous step are subjected to the clustering process, which gradually converges upwards until a group of highly condensed center point vectors are formed in the uppermost level. The center point vectors in the uppermost level are obtained after multiple iterative clusterings, which represent the highest level of abstraction of the original feature vector data and can effectively cover and summarize the data distribution characteristics in the database.

[0071] Then, after the center point vectors are screened, nodes corresponding to each center point vector are constructed, and the similarity between the center point vectors is calculated, and the connection relationship between the nodes is determined based on the similarity between the center point vectors. The calculation of the similarity is usually based on the cosine similarity, Euclidean distance or Manhattan distance, etc. The center point vectors with high similarity will be connected to form edges in the graph, which reflects the correlation and structured information between the data.

[0072] Finally, based on the nodes of the center point vectors and the edges between the nodes, a graph structure is constructed. The graph structure is a complex data network, where each node represents a center point vector, and the edges represent the association or similarity between the nodes (vectors).

[0073] By constructing the graph structure, similar center point vectors can be quickly located and queried, and then the clusters represented by the center point vectors and the vectors within the clusters can be found, which greatly improves the efficiency and accuracy of the search. In the subsequent search process, relevant center points can be quickly found according to the input query vector, and then the specific data vectors can be located through the center points, thereby realizing the efficient search of massive data. Therefore, by constructing the graph structure of the center vector library using the data screening algorithm, the search speed is improved, and the quality of the search results is optimized.

[0074] Optionally, in the multi-modal data retrieval method provided in Embodiment One of the present application, after determining the number of retrieval results according to the similarity, the above method further comprises: in the case that the number of retrieval results is less than or equal to the retrieval number threshold, retrieving the target feature vector in the database to be retrieved to obtain a second retrieval list; sorting the retrieval results in the second retrieval list according to the cosine similarity to obtain a sorted second retrieval list; calculating the confidence of each retrieval result in the sorted second retrieval list according to the preset retrieval threshold, filtering the retrieval results with a confidence lower than the preset confidence threshold, and obtaining a target retrieval list.

[0075] In Embodiment One, when it is detected that the number of preliminary retrieval results does not exceed the preset retrieval number threshold, full retrieval can be directly performed in the database to be retrieved, that is, the target feature vector and the feature vector corresponding to each data to be retrieved in the complete database to be retrieved are comprehensively matched. The retrieval result set produced by this matching process is called a second retrieval list. The second retrieval list contains all database records with a certain similarity to the target feature vector.

[0076] Then, the cosine similarity of each retrieval result in the second retrieval list is calculated. The higher the similarity value, the closer the angle between the two vectors in the vector space, that is, the more similar the data they represent. All retrieval results will be sorted according to their cosine similarity with the target feature vector to form a sorted second retrieval list, so as to ensure that the most relevant data entries are given priority.

[0077] Finally, in order to further improve the accuracy of the retrieval results, the sorted second retrieval list can be screened according to the preset confidence threshold. The confidence threshold serves as a standard for judging whether the retrieval result is sufficiently reliable, which is usually set based on business needs and experience data. Retrieval results with a confidence score lower than the preset threshold will be removed from the list. Finally, a more accurate target retrieval list with low-confidence items removed is output.

[0078] Through these steps, the retrieval strategy can be flexibly adjusted in the case of small retrieval result size, the quality of the retrieval results is guaranteed, and fast and accurate retrieval results are obtained.

[0079] Optionally, in the multi-modal data retrieval method provided in Embodiment One of the present application, the confidence of each retrieval result in the first retrieval list is calculated according to a preset retrieval threshold, and retrieval results with a confidence lower than the preset confidence threshold are filtered to obtain a target retrieval list, including: normalizing the confidence of each retrieval result in the first retrieval list according to the preset retrieval threshold to obtain the processed confidence of each retrieval result; determining retrieval results with a confidence higher than the preset confidence threshold in the first retrieval list to obtain the target retrieval list.

[0080] In Embodiment One, in order to adjust the confidence values of different retrieval results to the same scale, eliminate the dimensional influence, and ensure the fairness and effectiveness of the confidence comparison, the original confidence values can be converted to between 0 and 1 by using a mapping rule such as a linear function or a logarithmic function, and the confidence of each retrieval result in the first retrieval list is normalized according to the preset retrieval threshold, so that the credibility of each result can be more intuitively evaluated.

[0081] For example, the user inputs a preset retrieval threshold, uses a confidence mapping rule to establish a mapping relationship between the retrieval threshold and the confidence, performs normalization processing, and filters data that does not meet the retrieval threshold. The confidence mapping rule can be expressed as: Ta = Ti*(Tmax-Tmin)+Tmin, where Ta represents the processed confidence of each retrieval result described above, Ti is the preset retrieval threshold described above, Tmax is the maximum confidence among the confidences corresponding to all retrieval results in the first retrieval list, and Tmin is the minimum confidence.

[0082] Then, retrieval results with a confidence higher than the preset confidence threshold are screened in the first retrieval list, and these retrieval results with a high confidence are summarized to form a target retrieval list. The target retrieval list not only contains the most relevant and most credible data entries, but also may be accompanied by detailed confidence information of each data entry, providing the user with more comprehensive and more detailed retrieval results.

[0083] Through the above steps, the quality and credibility of the retrieval results are ensured, the user's demand for finding accurate information in a large amount of data is met, and the technical effect of improving the quality of the retrieval results is achieved.

[0084] Optionally, in Embodiment One, Figure 4is a flowchart of an optional multi-modal data retrieval method according to the embodiment one of the present application. First, the user inputs a natural language description or uploads a reference picture through the interface and sets a retrieval threshold T. After receiving these information, the multi-modal large model quickly extracts the vector features of the input information, and converts the natural language or picture into a digital expression that can be understood and processed by the system. The extracted vector features are mapped to a shared vector space, which contains the vector representation of all data to be retrieved, facilitating subsequent vector calculation and data retrieval. Then, using the cosine similarity calculation rule, the similarity between the target feature vector and the vector stored in the database is evaluated, and the number of preliminary retrieval results R is output. It is judged whether the number of retrieval results R exceeds the preset retrieval number threshold S.

[0085] If the number of retrieval results R > the number of retrieval thresholds S, it means that the retrieval results are too many, and may contain a large amount of irrelevant or low-quality data. At this time, the number of center points is customized, and the center point vector library is constructed by combining the GRAPH_PQ vector index algorithm. The most representative vectors are selected as the center points by clustering or random sampling to optimize the retrieval performance. Subsequently, using the IVF (Inverted File) index algorithm, efficient retrieval is performed in combination with the center point vector library, which narrows down the range of candidate data and speeds up the retrieval speed. Finally, the retrieval results sorted by cosine similarity are returned and normalized, that is, a mapping relationship is established based on the set retrieval threshold T and the vector confidence Ta, and the retrieval results with confidence lower than the threshold are filtered out, and finally a high-quality retrieval result set is formed, that is, the above-mentioned target retrieval list.

[0086] If the number of retrieval results R ≤ the number of retrieval thresholds S, the full retrieval method is directly used to match the retrieval statement with the vector library to obtain all related data. Similarly, the retrieval results sorted by cosine similarity are returned and normalized, that is, the mapping relationship between the user-set retrieval threshold T and the vector confidence Ta is used to filter the data, to ensure that the final returned retrieval results all meet or exceed the preset confidence threshold, forming a high-quality retrieval result set, that is, the above-mentioned target retrieval list.

[0087] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0088] Embodiment two

[0089] The embodiment two of the present application further provides a multi-modal data retrieval device. It should be noted that the multi-modal data retrieval device of the embodiment two of the present application can be used to execute the multi-modal data retrieval method provided in the embodiment one of the present application. The multi-modal data retrieval device provided in the embodiment two of the present application is introduced as follows.

[0090] Figure 5 FIG. 1 is a schematic diagram of the multi-modal data retrieval device provided in the embodiment two of the present application. As shown in the figure, the device comprises a conversion unit 501, a calculation unit 502, a first retrieval unit 503 and a first filtering unit 504. Figure 5

[0091] Specifically, the conversion unit 501 is configured to convert multi-modal data input by a user into a target feature vector by using a multi-modal large model, wherein the multi-modal data comprises at least one of the following: text data, image data.

[0092] The calculation unit 502 is configured to calculate the similarity between the target feature vector and a to-be-retrieved vector based on a cosine similarity formula, and determine the number of retrieval results according to the similarity, wherein the to-be-retrieved vector is a feature vector obtained by performing feature extraction on data in a to-be-retrieved database by using the multi-modal large model.

[0093] The first retrieval unit 503 is configured to, when the number of retrieval results is greater than a retrieval number threshold, perform retrieval on the target feature vector in the center vector library based on an inverted index algorithm to obtain a first retrieval list, wherein the first retrieval list at least comprises a plurality of retrieval results and a confidence degree corresponding to each retrieval result.

[0094] The first filtering unit 504 is configured to calculate the confidence degree corresponding to each retrieval result in the first retrieval list according to a preset retrieval threshold, filter retrieval results with a confidence degree lower than a preset confidence threshold, and obtain a target retrieval list.

[0095] ​The retrieval device for multi-modal data provided in the second embodiment of the application, through the conversion unit 501, converts the multi-modal data input by the user into a target feature vector through a multi-modal large model, wherein the multi-modal data includes at least one of the following: text data, image data; the calculation unit 502 calculates the similarity between the target feature vector and the to-be-retrieved vector based on the cosine similarity formula, and determines the number of retrieval results according to the similarity, wherein the to-be-retrieved vector is a feature vector obtained by performing feature extraction on the data in the to-be-retrieved database using the multi-modal large model; the first retrieval unit 503, in the case that the number of retrieval results is greater than the retrieval number threshold, performs retrieval on the target feature vector in the center vector library based on the inverted index algorithm, and obtains a first retrieval list, wherein the first retrieval list at least includes: a plurality of retrieval results, and a confidence degree corresponding to each retrieval result; the first filtering unit 504 calculates the confidence degree corresponding to each retrieval result in the first retrieval list according to the preset retrieval threshold, filters the retrieval results with a confidence degree lower than the preset confidence threshold, and obtains a target retrieval list, thereby solving the problem that in the related art, when image data or text data is retrieved, the retrieval efficiency is low due to a large amount of data.

[0096] The multi-modal large model is used to convert the multi-modal information such as text data and image data input by the user into a target feature vector, which can unify the data representation form, so that different types of retrieval requests can be efficiently processed in the same framework, and the technical effect of cross-modal information fusion retrieval is achieved. Meanwhile, by calculating the cosine similarity between the target feature vector and the to-be-retrieved vector in the database, the similarity between the data can be quickly evaluated, and further, when the number of retrieval results exceeds the preset threshold, the center point index algorithm is used for deep retrieval in the center vector library, and the first retrieval list obtained not only contains a plurality of retrieval results, but also adds the confidence degree of each result, which can greatly reduce the retrieval range and improve the response speed of retrieval and user experience. In addition, by calculating and filtering the confidence degree of each retrieval result, the entries with a confidence degree lower than the threshold are removed, and the final target retrieval list is generated, which ensures the high accuracy and reliability of the retrieval results, realizes high-quality data retrieval, further achieves the technical effect of improving the retrieval efficiency and performance, and also reduces the cost of processing invalid data, promotes the efficient use of data.

[0097] Optionally, in the multi-modal data retrieval device provided in Embodiment Two of the present application, the conversion unit 501 comprises: a generation sub-unit configured to receive multi-modal data input by a user and generate a retrieval sentence according to the multi-modal data; wherein the multi-modal data further comprises: retrieval information, the retrieval information comprising at least one of: data source, data generation time, preset retrieval threshold, data type; a first extraction sub-unit configured to input the multi-modal data into a multi-modal large model, extract a first feature vector of image data through an image encoder in the multi-modal large model, and extract a second feature vector of text data through a text encoder in the multi-modal large model; a second extraction sub-unit configured to extract a third feature vector of the retrieval sentence through the text encoder in the multi-modal large model; and a storage sub-unit configured to store the first feature vector, the second feature vector and the third feature vector into a preset storage space to obtain a target feature vector.

[0098] Optionally, in the multi-modal data retrieval device provided in Embodiment Two of the present application, the first retrieval unit 503 comprises: a classification sub-unit configured to classify the center vectors in the center vector library to obtain a plurality of clusters and create index information of each cluster in the plurality of clusters; a first determination sub-unit configured to determine a target cluster corresponding to the target feature vector according to the index information of each cluster; and a retrieval sub-unit configured to retrieve a feature vector similar to the target feature vector in the target cluster to obtain a first retrieval list.

[0099] Optionally, in the multi-modal data retrieval device provided in Embodiment Two of the present application, the device further comprises: a first determination unit configured to determine the number of center points according to the data volume of the to-be-retrieved database before retrieving the target feature vector in the center vector library based on the inverted index algorithm to obtain the first retrieval list; a dimension reduction unit configured to perform dimension reduction processing and compression processing on the feature vectors in the to-be-retrieved database through a priority queue algorithm to obtain processed to-be-retrieved feature vectors; a construction unit configured to filter center point vectors from the processed to-be-retrieved feature vectors based on the number of center points and construct a graph structure according to the center point vectors; and a second determination unit configured to determine the center vector library according to the graph structure.

[0100] Optionally, in the multi-modal data retrieval device provided in Embodiment Two of the present application, the construction unit comprises: a filtering sub-unit configured to filter the center point vectors from the processed to-be-retrieved feature vectors based on the number of center points through a data filtering algorithm, wherein the data filtering algorithm comprises at least one of: a clustering algorithm, a random sampling algorithm; a calculation sub-unit configured to construct nodes corresponding to the center point vectors and calculate the similarity between the center point vectors; a first construction sub-unit configured to construct edges between the nodes according to the similarity between the center point vectors; and a second construction sub-unit configured to construct the graph structure according to the nodes corresponding to the center point vectors and the edges between the nodes.

[0101] Optionally, in the multimodal data retrieval device provided in Example 2 of the present application, the above-mentioned device also includes: a second retrieval unit, which is used to determine the number of retrieval results based on the similarity, and when the number of retrieval results is less than or equal to the retrieval number threshold, search for the target feature vector in the database to be retrieved to obtain a second retrieval list; a sorting unit, which is used to sort the retrieval results in the second retrieval list according to the size of the cosine similarity to obtain a sorted second retrieval list; a second filtering unit, which is used to calculate the confidence corresponding to each retrieval result in the sorted second retrieval list based on a preset retrieval threshold, filter the retrieval results whose confidence is lower than the preset confidence threshold, and obtain a target retrieval list.

[0102] Optionally, in the multimodal data retrieval device provided in Example 2 of the present application, the above-mentioned first filtering unit 504 includes: a processing subunit, used to normalize the confidence corresponding to each retrieval result in the first retrieval list according to a preset retrieval threshold, and obtain the confidence of each retrieval result after processing; a second determination subunit, used to determine the retrieval results in the first retrieval list whose confidence is higher than the preset confidence threshold, and obtain a target retrieval list.

[0103] The multimodal data retrieval device includes a processor and a memory. The above-mentioned conversion unit 501, calculation unit 502, first retrieval unit 503 and first filtering unit 504 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to implement corresponding functions.

[0104] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to improve the search efficiency.

[0105] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0106] A third embodiment of the present invention provides a computer-readable storage medium having a program stored thereon, which implements a method for retrieving multimodal data when executed by a processor.

[0107] A fourth embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes a method for retrieving multimodal data when running.

[0108] Figure 6 Schematic diagram of an electronic device for retrieving multimodal data according to the fifth embodiment of the present application. Figure 6As shown, the fifth embodiment of the present application provides an electronic device, the device comprising a processor, a memory, and a program stored on the memory and executable on the processor, and the processor implements the following steps when executing the program: converting multi-modal data input by a user into a target feature vector through a multi-modal large model, wherein the multi-modal data comprises at least one of the following: text data, image data; calculating the similarity between the target feature vector and a to-be-retrieved vector based on a cosine similarity formula, and determining the number of retrieval results according to the similarity, wherein the to-be-retrieved vector is a feature vector obtained by performing feature extraction on data in a to-be-retrieved database using the multi-modal large model; in the case that the number of retrieval results is greater than a retrieval number threshold, performing retrieval on the target feature vector in a center vector library based on an inverted index algorithm to obtain a first retrieval list, wherein the first retrieval list at least includes: a plurality of retrieval results, and a confidence degree corresponding to each retrieval result; calculating the confidence degree corresponding to each retrieval result in the first retrieval list according to a preset retrieval threshold, filtering retrieval results with a confidence degree lower than the preset confidence threshold, and obtaining a target retrieval list.

[0109] The processor further implements the following steps when executing the program: converting multi-modal data input by a user into a target feature vector through a multi-modal large model, comprising: receiving multi-modal data input by a user, and generating a retrieval statement according to the multi-modal data; wherein the multi-modal data further comprises: retrieval information, the retrieval information comprising at least one of the following: data source, data generation time, preset retrieval threshold, and data type; inputting the multi-modal data into the multi-modal large model, extracting a first feature vector of image data through an image encoder in the multi-modal large model, and extracting a second feature vector of text data through a text encoder in the multi-modal large model; extracting a third feature vector of the retrieval statement through the text encoder in the multi-modal large model; storing the first feature vector, the second feature vector, and the third feature vector to a preset storage space to obtain the target feature vector.

[0110] The processor further implements the following steps when executing the program: performing retrieval on the target feature vector in a center vector library based on an inverted index algorithm to obtain a first retrieval list, comprising: classifying center vectors in the center vector library to obtain a plurality of clusters, and creating index information of each cluster in the plurality of clusters; determining a target cluster corresponding to the target feature vector according to the index information of each cluster; and retrieving feature vectors similar to the target feature vector in the target cluster to obtain the first retrieval list.

[0111] The processor further implements the following steps when executing the program: before searching the target feature vector in the center vector library based on the inverted index algorithm to obtain a first search list, the method further comprises: determining the number of center points according to the data volume of the database to be searched; performing dimension reduction processing and compression processing on the feature vectors in the database to be searched through a priority queue algorithm to obtain processed feature vectors to be searched; filtering center point vectors from the processed feature vectors to be searched based on the number of center points, and constructing a graph structure according to the center point vectors; and determining the center vector library according to the graph structure.

[0112] The processor further implements the following steps when executing the program: filtering center point vectors from the processed feature vectors to be searched based on the number of center points, and constructing a graph structure according to the center point vectors, comprising: filtering center point vectors from the processed feature vectors to be searched based on the number of center points through a data filtering algorithm, wherein the data filtering algorithm comprises at least one of a clustering algorithm and a random sampling algorithm; constructing nodes corresponding to the center point vectors and calculating the similarity between the center point vectors; constructing edges between the nodes according to the similarity between the center point vectors; and constructing the graph structure according to the nodes corresponding to the center point vectors and the edges between the nodes.

[0113] The processor further implements the following steps when executing the program: after determining the number of search results according to the similarity, the method further comprises: in the case that the number of search results is less than or equal to a search quantity threshold, searching the target feature vector in the database to be searched to obtain a second search list; sorting the search results in the second search list according to the cosine similarity to obtain a sorted second search list; calculating the confidence of each search result in the sorted second search list according to a preset search threshold, filtering the search results with a confidence lower than a preset confidence threshold to obtain a target search list.

[0114] The processor further implements the following steps when executing the program: calculating the confidence of each search result in the first search list according to a preset search threshold, filtering the search results with a confidence lower than a preset confidence threshold to obtain a target search list, comprising: performing normalization processing on the confidence of each search result in the first search list according to the preset search threshold to obtain the processed confidence of each search result; and determining the search results with a confidence higher than the preset confidence threshold in the first search list to obtain the target search list.

[0115] The device herein can be a server, a PC, a PAD, a mobile phone, etc.

[0116] The application also provides a computer program product suitable for executing a program initializing the following method steps when executed on a data processing device: converting multi-modal data input by a user into a target feature vector through a multi-modal large model, wherein the multi-modal data includes at least one of the following: text data, image data; calculating the similarity between the target feature vector and a to-be-retrieved vector based on a cosine similarity formula, and determining the number of retrieval results according to the similarity, wherein the to-be-retrieved vector is a feature vector obtained by performing feature extraction on data in a to-be-retrieved database using the multi-modal large model; in the case that the number of retrieval results is greater than a retrieval number threshold, retrieving the target feature vector in a center vector library based on an inverted index algorithm to obtain a first retrieval list, wherein the first retrieval list at least includes: a plurality of retrieval results, and a confidence degree corresponding to each retrieval result; calculating the confidence degree corresponding to each retrieval result in the first retrieval list according to a preset retrieval threshold, filtering retrieval results with a confidence degree lower than the preset confidence threshold, and obtaining a target retrieval list.

[0117] When executed on a data processing device, the program is also suitable for initializing the following method steps: converting multi-modal data input by a user into a target feature vector through a multi-modal large model, including: receiving multi-modal data input by a user, and generating a retrieval statement according to the multi-modal data; wherein the multi-modal data further includes: retrieval information, and the retrieval information includes at least one of the following: data source, data generation time, preset retrieval threshold, and data type; inputting the multi-modal data into the multi-modal large model, extracting a first feature vector of image data through an image encoder in the multi-modal large model, and extracting a second feature vector of text data through a text encoder in the multi-modal large model; extracting a third feature vector of the retrieval statement through the text encoder in the multi-modal large model; storing the first feature vector, the second feature vector, and the third feature vector to a preset storage space to obtain the target feature vector.

[0118] When executed on a data processing device, the program is also suitable for initializing the following method steps: retrieving the target feature vector in a center vector library based on an inverted index algorithm to obtain a first retrieval list, including: classifying center vectors in the center vector library to obtain a plurality of clusters, and creating index information of each cluster in the plurality of clusters; determining a target cluster corresponding to the target feature vector according to the index information of each cluster; retrieving feature vectors similar to the target feature vector in the target cluster to obtain the first retrieval list.

[0119] When executed on the data processing device, the program is also adapted to execute the method steps of: before searching the target feature vector in the center vector library based on the inverted index algorithm to obtain a first search list, the method further comprises: determining the number of center points according to the data volume of the database to be searched; performing dimension reduction processing and compression processing on the feature vectors in the database to be searched by a priority queue algorithm to obtain processed feature vectors to be searched; filtering the center point vectors from the processed feature vectors to be searched based on the number of center points, and constructing a graph structure according to the center point vectors; and determining the center vector library according to the graph structure.

[0120] When executed on the data processing device, the program is also adapted to execute the method steps of: filtering the center point vectors from the processed feature vectors to be searched based on the number of center points, and constructing a graph structure according to the center point vectors, comprising: filtering the center point vectors from the processed feature vectors to be searched based on the number of center points by a data filtering algorithm, wherein the data filtering algorithm comprises at least one of: a clustering algorithm, a random sampling algorithm; constructing nodes corresponding to the center point vectors, and calculating the similarity between the center point vectors; constructing edges between the nodes according to the similarity between the center point vectors; and constructing the graph structure according to the nodes corresponding to the center point vectors and the edges between the nodes.

[0121] When executed on the data processing device, the program is also adapted to execute the method steps of: after determining the number of search results according to the similarity, the method further comprises: in the case that the number of search results is less than or equal to a search quantity threshold, searching the target feature vector in the database to be searched to obtain a second search list; sorting the search results in the second search list according to the cosine similarity to obtain a sorted second search list; calculating the confidence of each search result corresponding to the sorted second search list according to a preset search threshold, filtering the search results with a confidence lower than a preset confidence threshold to obtain a target search list.

[0122] When executed on the data processing device, the program is also adapted to execute the method steps of: calculating the confidence of each search result corresponding to the first search list according to a preset search threshold, filtering the search results with a confidence lower than a preset confidence threshold to obtain a target search list, comprising: normalizing the confidence of each search result corresponding to the first search list according to a preset search threshold to obtain the processed confidence of each search result; determining the search results with a confidence higher than a preset confidence threshold in the first search list to obtain the target search list.

[0123] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0124] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0125] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0126] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0127] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0128] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile or non-volatile random access memory (RAM), among others. The memory is an example of computer readable media.

[0129] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0130] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0131] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A multimodal data retrieval method, characterized in that: include: Converting multimodal data input by a user into a target feature vector through a multimodal macro model, wherein the multimodal data includes at least one of the following: text data and image data; Calculating the similarity between the target feature vector and the vector to be searched based on a cosine similarity formula, and determining the number of search results based on the similarity, wherein the vector to be searched is a feature vector obtained by extracting features from the data in the database to be searched using the multimodal large model; When the number of the search results is greater than the search number threshold, searching the target feature vector in the central vector library based on the inverted index algorithm to obtain a first search list, wherein the first search list at least includes: multiple search results and a confidence level corresponding to each search result; The confidence level corresponding to each search result in the first search list is calculated according to a preset search threshold, and search results with a confidence level lower than the preset confidence threshold are filtered to obtain a target search list.

2. The method according to claim 1, characterized in that The multimodal data input by the user is converted into a target feature vector through a multimodal large model, including: Receive multimodal data input by a user and generate a search statement based on the multimodal data; wherein the multimodal data also includes: search information, the search information including at least one of the following: data source, data generation time, the preset search threshold, and data type; Inputting the multimodal data into the multimodal large model, extracting a first eigenvector of the image data through an image encoder in the multimodal large model, and extracting a second eigenvector of the text data through a text encoder in the multimodal large model; Extracting a third feature vector of the search sentence by a text encoder in the multimodal large model; The first eigenvector, the second eigenvector, and the third eigenvector are stored in a preset storage space to obtain the target eigenvector.

3. The method according to claim 1, characterized in that The target feature vector is searched in the central vector library based on the inverted index algorithm to obtain a first search list, including: Classifying the central vectors in the central vector library to obtain multiple clusters, and creating index information for each cluster in the multiple clusters; Determining a target cluster corresponding to the target feature vector according to the index information of each cluster; Feature vectors similar to the target feature vector are searched in the target cluster to obtain the first search list.

4. The method according to claim 1, wherein Before searching the target feature vector in the central vector library based on the inverted index algorithm to obtain the first search list, the method further includes: Determining the number of center points according to the amount of data in the database to be searched; Performing dimensionality reduction and compression processing on the feature vectors in the database to be retrieved by using a priority queue algorithm to obtain processed feature vectors to be retrieved; Filtering center point vectors of the processed feature vectors to be retrieved based on the number of center points, and constructing a graph structure according to the center point vectors; The central vector library is determined according to the graph structure.

5. The method according to claim 4, characterized in that The method further comprises: selecting a central point vector from the processed feature vector to be retrieved based on the number of central points, and constructing a graph structure according to the central point vector, including: Filtering the center point vectors of the processed feature vectors to be retrieved based on the number of center points using a data screening algorithm, wherein the data screening algorithm includes at least one of the following: a clustering algorithm and a random sampling algorithm; Constructing nodes corresponding to the center point vectors and calculating similarities between the center point vectors; Constructing edges between the nodes based on the similarity between the center point vectors; The graph structure is constructed based on the nodes corresponding to the center point vector and the edges between the nodes.

6. The method according to claim 1, characterized in that After determining the number of search results based on the similarity, the method further includes: When the number of the search results is less than or equal to the search number threshold, searching the target feature vector in the database to be searched to obtain a second search list; sorting the search results in the second search list according to the cosine similarity to obtain a sorted second search list; The confidence level corresponding to each search result in the sorted second search list is calculated according to a preset search threshold, and search results with a confidence level lower than the preset confidence threshold are filtered to obtain the target search list.

7. The method according to claim 1, characterized in that Calculating the confidence level corresponding to each search result in the first search list according to a preset search threshold, filtering search results with a confidence level lower than the preset confidence threshold, and obtaining a target search list, including: Normalizing the confidence level corresponding to each search result in the first search list according to the preset search threshold to obtain a processed confidence level for each search result; The target search list is obtained by determining the search results whose confidence levels are higher than a preset confidence threshold in the first search list.

8. A multimodal data retrieval device, characterized in that: include: a conversion unit, configured to convert multimodal data input by a user into a target feature vector through a multimodal large model, wherein the multimodal data includes at least one of the following: text data, image data; a calculation unit, configured to calculate the similarity between the target feature vector and the vector to be searched based on a cosine similarity formula, and determine the number of search results based on the similarity, wherein the vector to be searched is a feature vector obtained by extracting features from data in the database to be searched using the multimodal large model; A first retrieval unit is configured to, when the number of retrieval results is greater than a retrieval number threshold, search the target feature vector in a central vector library based on an inverted index algorithm to obtain a first retrieval list, wherein the first retrieval list includes at least: a plurality of retrieval results and a confidence level corresponding to each retrieval result; The first filtering unit is used to calculate the confidence corresponding to each search result in the first search list according to a preset search threshold, filter the search results with confidence lower than the preset confidence threshold, and obtain a target search list.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes stored computer instructions, wherein when the computer instructions are executed by a processor, the multimodal data retrieval method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The device comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal data retrieval method according to any one of claims 1 to 7.

Images