A method and apparatus for data retrieval, and a network attached storage device

Abstract

The application relates to the technical field of data retrieval, and discloses a data retrieval method and device and a network-attached storage device, the method comprising the following steps: obtaining to-be-retrieved data, the to-be-retrieved data comprising at least one kind of to-be-retrieved sub-data; each kind of to-be-retrieved sub-data comprising one of text data, image data and video data; performing a feature extraction and fusion operation on the preprocessed to-be-retrieved data based on a feature extraction model to obtain a fusion feature vector corresponding to the to-be-retrieved data; determining a vector index corresponding to the fusion feature vector based on a target index structure corresponding to a vector database; and performing a retrieval operation on the vector index in the vector database based on a determined retrieval algorithm to obtain a retrieval result corresponding to the to-be-retrieved data. It can be seen that the application can improve the flexibility of data retrieval and is beneficial to providing more efficient retrieval services for users.

Description

Technical Field

[0001] This invention relates to the field of data retrieval technology, and in particular to a data retrieval method and apparatus, and a network attached storage device. Background Technology

[0002] Since users often need to retrieve data when browsing websites or using applications, data retrieval functions have gradually become a major feature of various websites and software, allowing users to easily obtain the information they need through retrieval.

[0003] Practice has shown that most existing search functions only achieve data retrieval through traditional text search methods. However, traditional search methods are difficult to handle retrieval needs for unstructured data (such as text, images, and audio). Therefore, it is particularly important to propose a technical solution that can improve the flexibility of data retrieval and provide users with more efficient search services. Summary of the Invention

[0004] This invention provides a data retrieval method and apparatus, and a network-attached storage device, which can improve the flexibility of data retrieval and facilitate the provision of more efficient retrieval services to users.

[0005] To address the aforementioned technical problems, the first aspect of this invention discloses a data retrieval method applied to a multimodal retrieval system, the multimodal retrieval system comprising a vector database and a pre-trained feature extraction model, the method comprising:

[0006] Acquire data to be retrieved, the data to be retrieved including at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data, and video data;

[0007] Based on the feature extraction model, a feature extraction and fusion operation is performed on the preprocessed data to be retrieved to obtain the fused feature vector corresponding to the data to be retrieved.

[0008] Based on the target index structure corresponding to the vector database, determine the vector index corresponding to the fused feature vector;

[0009] Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain the retrieval results corresponding to the data to be retrieved.

[0010] As an optional implementation, in the first aspect of the present invention, the feature extraction model includes at least one feature extraction sub-model, and each of the feature extraction sub-models has a mapping relationship with at least one of the data types;

[0011] The step of performing feature extraction and fusion operations on the preprocessed data to be retrieved based on the feature extraction model to obtain the fused feature vector corresponding to the data to be retrieved includes:

[0012] From the feature extraction model, determine the feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed data to be retrieved;

[0013] For each type of sub-data to be retrieved, based on the feature extraction sub-model, a feature extraction operation is performed on the sub-data to be retrieved to obtain the target feature vector corresponding to the sub-data to be retrieved.

[0014] Based on the target feature vectors corresponding to all the sub-data to be retrieved, determine the fused feature vector corresponding to the data to be retrieved.

[0015] As an optional implementation, in a first aspect of the present invention, determining the feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed data to be retrieved from the feature extraction model includes:

[0016] For each type of sub-data to be retrieved, when the sub-data to be retrieved is the image data or the video data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined to be the image feature extraction sub-model, and the first feature extraction sub-model adopts a self-attention mechanism.

[0017] For each type of sub-data to be retrieved, when the sub-data to be retrieved is the text data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined as the second feature extraction sub-model;

[0018] And, for each type of sub-data to be retrieved, based on the feature extraction sub-model, performing a feature extraction operation on the sub-data to be retrieved to obtain the target feature vector corresponding to the sub-data to be retrieved includes:

[0019] For each type of sub-data to be retrieved, when the sub-data to be retrieved is the image data or the video data, the semantic features corresponding to the sub-data to be retrieved are extracted based on the self-attention mechanism of the first feature extraction sub-model, and used as the target feature vector corresponding to the sub-data to be retrieved.

[0020] For each type of sub-data to be retrieved, when the sub-data to be retrieved is text data, the sentence-level features corresponding to the sub-data to be retrieved are extracted based on the second feature extraction sub-model and used as the target feature vector corresponding to the sub-data to be retrieved.

[0021] As an optional implementation, in the first aspect of the present invention, determining the fused feature vector corresponding to the data to be retrieved based on the target feature vectors corresponding to all the sub-data to be retrieved includes:

[0022] When the data to be retrieved includes only one type of sub-data to be retrieved, the target feature vector corresponding to the sub-data to be retrieved is determined as the fusion feature vector corresponding to the data to be retrieved.

[0023] When the data to be retrieved includes at least two types of sub-data to be retrieved, based on the multimodal fusion strategy corresponding to the multimodal retrieval system, feature fusion processing is performed on the target feature vectors corresponding to all the sub-data to be retrieved to obtain the fused feature vector corresponding to the data to be retrieved.

[0024] As an optional implementation, in the first aspect of the present invention, the method further includes:

[0025] Obtain the business scenario corresponding to the data to be retrieved and the retrieval requirements corresponding to the data to be retrieved;

[0026] And, based on the determined retrieval algorithm, performing a retrieval operation on the vector index in the vector database to obtain the retrieval results corresponding to the data to be retrieved includes:

[0027] Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain multiple candidate retrieval results;

[0028] Analyze the business relevance of each candidate search result to the business scenario;

[0029] Filter out at least one target correlation degree that is greater than or equal to the preset business correlation degree from all the said business correlation degrees;

[0030] Based on all the candidate search results corresponding to the target relevance, the candidate search results that meet the search requirements are determined as the search results corresponding to the data to be searched.

[0031] As an optional implementation, in the first aspect of the present invention, the method further includes:

[0032] Obtain the test vector dimension corresponding to the vector database and the test vector dataset corresponding to the test vector dimension;

[0033] Based on the test vector dataset corresponding to the test vector dimension, a retrieval operation on the test vector dataset is performed in the vector database to obtain the test retrieval results corresponding to the test vector dimension;

[0034] Analyze the test retrieval results to obtain the retrieval performance corresponding to the test vector dimension;

[0035] Based on the predetermined retrieval performance standard and the retrieval performance, the test vector dimension is adjusted to obtain the target vector dimension corresponding to the vector database;

[0036] And, determining the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database includes:

[0037] Based on the target index structure corresponding to the vector database, the fused feature vector is transformed into a vector index corresponding to the target vector dimension.

[0038] As an optional implementation, in the first aspect of the present invention, the method further includes:

[0039] Obtain the multimodal dataset to be stored, wherein the multimodal dataset includes multiple sets of multimodal data;

[0040] Based on the data compression interface corresponding to the multimodal retrieval system, a compression operation is performed on the multimodal dataset to obtain a set of compressed files corresponding to the multimodal dataset, wherein the set of compressed files includes multiple compressed files;

[0041] Extract the compressed file queue from the set of compressed files;

[0042] Determine whether the compressed file queue meets the preset data processing conditions;

[0043] When it is determined that the compressed file queue meets the preset data processing conditions, a batch processing operation is performed on the compressed file queue based on the inference interface corresponding to the multimodal retrieval system to obtain the vector dataset to be stored corresponding to the target vector dimension.

[0044] The vector dataset to be stored is stored in the vector database.

[0045] A second aspect of the present invention discloses a data retrieval apparatus applied to a multimodal retrieval system, the multimodal retrieval system comprising a vector database and a pre-trained feature extraction model, the apparatus comprising:

[0046] The acquisition module is used to acquire data to be retrieved, the data to be retrieved including at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data and video data.

[0047] The feature extraction and fusion module is used to perform feature extraction and fusion operations on the preprocessed data to be retrieved based on the feature extraction model, so as to obtain the fused feature vector corresponding to the data to be retrieved.

[0048] The determining module is used to determine the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database;

[0049] The retrieval module is used to perform a retrieval operation on the vector index in the vector database based on a determined retrieval algorithm, and obtain the retrieval results corresponding to the data to be retrieved.

[0050] As an optional implementation, in a second aspect of the present invention, the feature extraction model includes at least one feature extraction sub-model, and each of the feature extraction sub-models has a mapping relationship with at least one of the data types;

[0051] The feature extraction and fusion module performs feature extraction and fusion operations on the preprocessed data to be retrieved based on the feature extraction model to obtain the fused feature vector corresponding to the data to be retrieved. The specific methods include:

[0052] From the feature extraction model, determine the feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed data to be retrieved;

[0053] For each type of sub-data to be retrieved, based on the feature extraction sub-model, a feature extraction operation is performed on the sub-data to be retrieved to obtain the target feature vector corresponding to the sub-data to be retrieved.

[0054] Based on the target feature vectors corresponding to all the sub-data to be retrieved, determine the fused feature vector corresponding to the data to be retrieved.

[0055] As an optional implementation, in a second aspect of the present invention, the specific method by which the feature extraction fusion module determines the feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed data to be retrieved from the feature extraction model includes:

[0056] For each type of sub-data to be retrieved, when the sub-data to be retrieved is the image data or the video data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined to be the image feature extraction sub-model, and the first feature extraction sub-model adopts a self-attention mechanism.

[0057] For each type of sub-data to be retrieved, when the sub-data to be retrieved is the text data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined as the second feature extraction sub-model;

[0058] Furthermore, the specific methods by which the feature extraction and fusion module performs feature extraction operations on each type of sub-data to be retrieved, based on the feature extraction sub-model, to obtain the target feature vector corresponding to the sub-data to be retrieved include:

[0059] For each type of sub-data to be retrieved, when the sub-data to be retrieved is the image data or the video data, the semantic features corresponding to the sub-data to be retrieved are extracted based on the self-attention mechanism of the first feature extraction sub-model, and used as the target feature vector corresponding to the sub-data to be retrieved.

[0060] For each type of sub-data to be retrieved, when the sub-data to be retrieved is text data, the sentence-level features corresponding to the sub-data to be retrieved are extracted based on the second feature extraction sub-model and used as the target feature vector corresponding to the sub-data to be retrieved.

[0061] As an optional implementation, in the second aspect of the present invention, the specific method by which the feature extraction and fusion module determines the fused feature vector corresponding to the data to be retrieved based on the target feature vectors corresponding to all the sub-data to be retrieved includes:

[0062] When the data to be retrieved includes only one type of sub-data to be retrieved, the target feature vector corresponding to the sub-data to be retrieved is determined as the fusion feature vector corresponding to the data to be retrieved.

[0063] When the data to be retrieved includes at least two types of sub-data to be retrieved, based on the multimodal fusion strategy corresponding to the multimodal retrieval system, feature fusion processing is performed on the target feature vectors corresponding to all the sub-data to be retrieved to obtain the fused feature vector corresponding to the data to be retrieved.

[0064] As an optional implementation, in a second aspect of the present invention, the acquisition module is further configured to acquire the business scenario corresponding to the data to be retrieved and the retrieval requirements corresponding to the data to be retrieved;

[0065] Furthermore, the specific methods by which the retrieval module performs a retrieval operation on the vector index in the vector database based on the determined retrieval algorithm to obtain the retrieval results corresponding to the data to be retrieved include:

[0066] Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain multiple candidate retrieval results;

[0067] Analyze the business relevance of each candidate search result to the business scenario;

[0068] Filter out at least one target correlation degree that is greater than or equal to the preset business correlation degree from all the said business correlation degrees;

[0069] Based on all the candidate search results corresponding to the target relevance, the candidate search results that meet the search requirements are determined as the search results corresponding to the data to be searched.

[0070] As an optional implementation, in the second aspect of the present invention, the acquisition module is further configured to acquire the test vector dimension corresponding to the vector database and the test vector dataset corresponding to the test vector dimension;

[0071] The retrieval module is further configured to perform a retrieval operation on the test vector dataset in the vector database based on the test vector dataset corresponding to the test vector dimension, and obtain the test retrieval result corresponding to the test vector dimension;

[0072] The device further includes:

[0073] The analysis module is used to analyze the test retrieval results and obtain the retrieval performance corresponding to the test vector dimension;

[0074] The adjustment module is used to adjust the test vector dimension based on a predetermined retrieval performance standard and the retrieval performance, so as to obtain the target vector dimension corresponding to the vector database.

[0075] Furthermore, the specific method by which the determining module determines the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database includes:

[0076] Based on the target index structure corresponding to the vector database, the fused feature vector is transformed into a vector index corresponding to the target vector dimension.

[0077] As an optional implementation, in a second aspect of the present invention, the acquisition module is further configured to acquire a multimodal dataset to be stored, the multimodal dataset comprising multiple sets of multimodal data;

[0078] The device further includes:

[0079] The compression module is used to perform compression operations on the multimodal dataset based on the data compression interface corresponding to the multimodal retrieval system, to obtain a set of compressed files corresponding to the multimodal dataset, wherein the set of compressed files includes multiple compressed files;

[0080] The extraction module is used to extract a queue of compressed files from the set of compressed files;

[0081] The judgment module is used to determine whether the compressed file queue meets the preset data processing conditions;

[0082] The batch processing module is used to perform batch processing operations on the compressed file queue based on the inference interface corresponding to the multimodal retrieval system when the judgment module determines that the compressed file queue meets the preset data processing conditions, so as to obtain the vector dataset to be stored corresponding to the target vector dimension.

[0083] A storage module is used to store the vector dataset to be stored into the vector database.

[0084] A third aspect of the present invention discloses another network-attached storage device, the network-attached storage device comprising:

[0085] Memory containing executable program code;

[0086] A processor coupled to the memory;

[0087] The processor calls the executable program code stored in the memory to execute the data retrieval method disclosed in the first aspect of the present invention.

[0088] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the data retrieval method disclosed in the first aspect of the present invention.

[0089] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0090] Implementing this invention enables the extraction and fusion of features from the preprocessed data after acquisition, based on a feature extraction model, to obtain a fused feature vector. Then, based on the target index structure of the vector database, the vector index corresponding to the fused feature vector is determined. Finally, based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain the retrieval results. This invention enables vector retrieval of multimodal data based on a vector database and a feature extraction model, achieving multimodal data retrieval based on data semantic content. This improves the flexibility and accuracy of unstructured data analysis, thereby enhancing the flexibility and accuracy of data retrieval and generating accurate and reliable retrieval results, ultimately providing users with more efficient retrieval services. Attached Figure Description

[0091] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0092] Figure 1 This is a flowchart illustrating a data retrieval method disclosed in an embodiment of the present invention;

[0093] Figure 2 This is a flowchart illustrating another data retrieval method disclosed in an embodiment of the present invention;

[0094] Figure 3 This is a schematic diagram of an index structure disclosed in an embodiment of the present invention;

[0095] Figure 4 This is a schematic diagram illustrating the business scenarios to which the data retrieval method disclosed in this embodiment of the invention is applicable;

[0096] Figure 5 This is a flowchart illustrating another data retrieval method disclosed in an embodiment of the present invention;

[0097] Figure 6 This is a flowchart illustrating another data retrieval method disclosed in an embodiment of the present invention;

[0098] Figure 7 This is a schematic diagram of the structure of a data retrieval device disclosed in an embodiment of the present invention;

[0099] Figure 8 This is a schematic diagram of the structure of another data retrieval device disclosed in an embodiment of the present invention;

[0100] Figure 9 This is a schematic diagram of the structure of a network-attached storage device disclosed in an embodiment of the present invention. Detailed Implementation

[0101] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0102] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0103] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0104] This invention discloses a data retrieval method and apparatus, and a network attached storage device. After acquiring the data to be retrieved, it performs feature extraction and fusion on the preprocessed data based on a feature extraction model to obtain a fused feature vector. Based on the target index structure of the vector database, it determines the vector index corresponding to the fused feature vector. Then, based on the determined retrieval algorithm, it performs a retrieval operation on the vector index in the vector database to obtain retrieval results. This invention enables vector retrieval of multimodal data based on a vector database and a feature extraction model, achieving multimodal data retrieval based on data semantic content. This improves the flexibility and accuracy of unstructured data analysis, thereby enhancing the flexibility and accuracy of data retrieval and generating accurate and reliable retrieval results, thus facilitating the provision of more efficient retrieval services to users. Detailed descriptions follow.

[0105] Example 1

[0106] Please see Figure 1 , Figure 1 This is a flowchart illustrating a data retrieval method disclosed in an embodiment of the present invention. Figure 1 The described data retrieval method can be applied to a data retrieval device, which may include one of a retrieval equipment, a retrieval terminal, and a server. The server may include a local server or a cloud server; this embodiment of the invention is not limited thereto. The device can be applied to a multimodal retrieval system, which includes a vector database and a pre-trained feature extraction model. The method can also be applied to a private cloud storage system; optionally, the private cloud storage system may be used with NAS services; this embodiment of the invention is not limited thereto. Figure 1 As shown, the data retrieval method may include the following operations:

[0107] 101. Obtain the data to be retrieved.

[0108] In this embodiment of the invention, the data to be retrieved includes at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data, and video data; optionally, the sub-data to be retrieved may also be other unstructured data, such as audio data, point cloud data, or user behavior data, which is not limited in this embodiment of the invention.

[0109] 102. Based on the feature extraction model, perform feature extraction and fusion operations on the preprocessed data to be retrieved to obtain the fused feature vector corresponding to the data to be retrieved.

[0110] In this embodiment of the invention, the feature extraction and fusion operation includes a feature extraction operation and a feature fusion operation.

[0111] 103. Based on the target index structure corresponding to the vector database, determine the vector index corresponding to the fused feature vector.

[0112] In this embodiment of the invention, the vector database can be FAISS (Facebook AI Similarity Search, an open-source vector database for clustering and similarity search developed by the Facebook AI team), or other vector databases; this embodiment of the invention does not limit the specific vector database. The target index structure can be a pre-defined index structure. Optionally, the target index structure can be a combined index structure composed of an HNSW (Hierarchical Navigable Small World) graph structure and PQ (Product Quantization). Using this combined index structure, efficient near-nearest neighbor search capabilities can be achieved through HNSW, and the storage pressure on the database can be reduced while maintaining retrieval accuracy through PQ. This enables the location of the most relevant retrieval structure in a large-scale dataset, improving retrieval efficiency. The target index structure can also be other index structures, such as IVF (Inverted File Index); this embodiment of the invention does not limit the specific index structure.

[0113] In this embodiment of the invention, for example, the index structure of a vector index can be as follows: Figure 3 As shown, Figure 3 This is a schematic diagram of an index structure disclosed in an embodiment of the present invention. Specifically, this structure can be used to retrieve video data containing image d, but the embodiment of the present invention does not limit it.

[0114] 104. Based on the determined retrieval algorithm, perform a retrieval operation on the vector index in the vector database to obtain the retrieval results corresponding to the data to be retrieved.

[0115] In this embodiment of the invention, optionally, the retrieval algorithm can be a deep learning-based query optimization algorithm, or other retrieval algorithms; this embodiment of the invention does not impose any limitations. Optionally, there may be one or more retrieval results with a relevance greater than a preset relevance to the data to be retrieved; this embodiment of the invention does not impose any limitations.

[0116] As can be seen, the method described in the embodiments of the present invention can, after acquiring the data to be retrieved, perform feature extraction and fusion on the preprocessed data to be retrieved based on the feature extraction model to obtain a fused feature vector. Based on the target index structure corresponding to the vector database, the vector index corresponding to the fused feature vector is determined. Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain the retrieval results. This method can realize vector retrieval for multimodal data based on the vector database and the feature extraction model, thereby achieving multimodal data retrieval based on data semantic content. This improves the flexibility and accuracy of unstructured data analysis, thereby improving the flexibility and accuracy of multimodal data analysis, and thus improving the flexibility and accuracy of data retrieval. In turn, it generates accurate and reliable retrieval results, which is conducive to providing users with more efficient retrieval services.

[0117] In an optional embodiment, the method may further include the following operations:

[0118] The data to be retrieved is compressed to obtain the compressed data to be retrieved.

[0119] For each type of sub-data in the compressed data to be retrieved, perform the preprocessing operation corresponding to the data type of the sub-data to be retrieved to obtain the preprocessed sub-data to be retrieved.

[0120] This approach, by compressing the data to be retrieved, reduces the scale of subsequent data processing, which is beneficial for improving data analysis efficiency and reducing the memory required for subsequent data retrieval, thereby improving data retrieval efficiency. Furthermore, by preprocessing the data to be retrieved, the accuracy and efficiency of subsequent feature extraction can be improved.

[0121] In this optional embodiment, optionally, for each type of sub-data to be retrieved in the compressed data to be retrieved, performing a preprocessing operation corresponding to the data type of the sub-data to be retrieved to obtain preprocessed sub-data to be retrieved may include the following operations:

[0122] For each type of sub-data to be retrieved in the compressed data to be retrieved, when the sub-data to be retrieved is image data, image quality enhancement processing is performed on the image data to obtain the enhanced image data.

[0123] When the sub-data to be retrieved is video data, the video data is decomposed into multiple video frames, and inter-frame correlation analysis is performed on all video frames to obtain the correlation analysis results; based on the correlation analysis results, at least one key frame is determined from all video frames to obtain the extracted video data.

[0124] When the sub-data to be retrieved is text data, semantic enhancement processing is performed on the text data to obtain semantically enhanced text data;

[0125] The preprocessed data to be retrieved includes at least one of image data enhanced by image enhancement, video data after frame extraction, and text data enhanced by semantic enhancement.

[0126] For example, image quality enhancement processing can be performed on image data using GANs (Generative Adversarial Networks) technology; semantic enhancement processing can be performed on text data using NLP (Natural Language Processing) technology. This embodiment of the invention does not limit the scope of the invention.

[0127] As can be seen, this optional embodiment can also select different preprocessing operations according to different data types, which can improve the flexibility and accuracy of data preprocessing, thereby improving the reliability of preprocessed data, and further improving the accuracy and efficiency of subsequent feature extraction.

[0128] In another alternative embodiment, the method may further include the following operations:

[0129] Obtain the business scenario corresponding to the data to be retrieved and the retrieval requirements corresponding to the data to be retrieved;

[0130] Furthermore, based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain the retrieval results corresponding to the data to be retrieved. This may include the following operations:

[0131] Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain multiple candidate retrieval results;

[0132] Analyze the business relevance of each candidate search result to the business scenario;

[0133] Filter out at least one target relevance that is greater than or equal to the preset business relevance from all business relevance scores;

[0134] Based on the candidate search results corresponding to all target relevance, the candidate search results that meet the search requirements are determined as the search results corresponding to the data to be searched.

[0135] Wherein, the business scenario is used to represent the business that needs to be implemented based on the search results corresponding to the data to be searched; wherein, optionally, the search requirements can be determined based on the business scenario, and this embodiment of the invention does not limit them; the search requirements may include one or more of the following combinations: the number of search results, the preset correlation between the search results and the data to be searched, the search time, and the search memory consumption, and this embodiment of the invention does not limit them.

[0136] For example, the business scenario corresponding to the data to be retrieved can be... Figure 4 The present invention does not limit the combination of one or more business scenarios shown; wherein... Figure 4 This is a schematic diagram illustrating the business scenarios to which the data retrieval method disclosed in this embodiment of the invention is applicable. Specifically, this method can be applied to one or more combinations of business scenario types such as recommendation type, image type, video type, audio type, NLP type, spatiotemporal type, and interdisciplinary type. This embodiment of the invention does not limit the application of this method.

[0137] As can be seen, this optional embodiment can perform a retrieval operation on the vector index in the vector database based on the determined retrieval algorithm to obtain multiple candidate retrieval results, analyze the business relevance between each candidate retrieval result and the business scenario, and then filter at least one target relevance greater than or equal to the preset business relevance from all business relevances. Based on the candidate retrieval results corresponding to the target relevance, the candidate retrieval results that meet the retrieval requirements are determined as the retrieval results. This can improve the accuracy of the association analysis of candidate retrieval results with the business scenario, thereby improving the flexibility and accuracy of retrieval result filtering, which in turn helps to make the filtered retrieval results more in line with the retrieval requirements, and further improve the accuracy and reliability of the retrieval results.

[0138] In this optional embodiment, when the business scenario corresponding to the data to be retrieved is a private cloud storage scenario, the method may further include:

[0139] Identify sensitive information corresponding to private cloud storage scenarios;

[0140] Determine whether any of the candidate search results contain sensitive information;

[0141] When it is determined that there are search results involving sensitive information among all candidate search results, the search results involving sensitive information are identified as the filtered search results;

[0142] Remove all filtered search results from all candidate search results to obtain filtered candidate search results; and trigger the execution of an operation to analyze the business relevance of each candidate search result to the business scenario.

[0143] Sensitive information may include one or more types of information that need to be protected, such as user personal information, system security information, and account password information. This embodiment of the invention does not limit the scope of such information.

[0144] As can be seen, this optional embodiment can also, when the business scenario is a private cloud storage scenario, determine whether the candidate search results involve sensitive information, remove the candidate search results that are determined to involve sensitive information, further screen the search results, improve the accuracy of the search results, thereby improving the information security provided to users by the search results, and helping to improve the data security of the multimodal search system.

[0145] In yet another alternative embodiment, the method may further include the following operations:

[0146] Obtain the test vector dimension corresponding to the vector database and the test vector dataset corresponding to the test vector dimension;

[0147] Based on the test vector dataset corresponding to the test vector dimension, perform a retrieval operation on the test vector dataset in the vector database to obtain the test retrieval results corresponding to the test vector dimension;

[0148] Analyze the test retrieval results to obtain the retrieval performance corresponding to the test vector dimension;

[0149] Based on the predetermined retrieval performance standards and retrieval performance, the test vector dimension is adjusted to obtain the target vector dimension corresponding to the vector database.

[0150] There may be one or more test vector dimensions. Optionally, when there are multiple test vector dimensions, the test vector dimension with the best retrieval performance can be selected from all test vector dimensions based on retrieval performance as the test vector dimension to be adjusted. Then, based on the retrieval performance standard and the retrieval performance corresponding to the test vector dimension to be adjusted, the test vector dimension is adjusted to obtain the target vector dimension corresponding to the vector database. This embodiment of the invention does not limit the scope of the invention.

[0151] The retrieval performance may include one or more of the following: retrieval time, retrieval speed, and retrieval memory consumption. This embodiment of the invention does not limit the performance. For example, the target vector dimension may be 576 or other values. This embodiment of the invention does not limit the performance.

[0152] As can be seen, this optional embodiment can test the data retrieval performance of the test vector dimension based on the test vector dataset corresponding to the test vector dimension. Based on the retrieval performance standard, the test vector dimension can be adjusted to obtain the target vector dimension. This can improve the accuracy of determining the vector dimension corresponding to the vector database, thereby improving the precision of the vector dimension corresponding to the vector database. This helps to reduce vector storage space while ensuring that the vector storage in the database has sufficient information, and improves data retrieval efficiency, which is conducive to generating retrieval results more efficiently.

[0153] In this optional embodiment, determining the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database may include the following operations:

[0154] Based on the target index structure corresponding to the vector database, the fused feature vector is transformed into a vector index corresponding to the target vector dimension.

[0155] For example, the above vector index can be a vector index with the structure composed of HNSW graph structure and PQ as the index structure and 576 as the vector dimension. This embodiment of the invention does not limit the scope of the invention.

[0156] As can be seen, this optional embodiment can also transform the fused feature vector into a vector index based on the target index result and the target vector dimension, which can improve the efficiency of data matching in the vector database based on the vector index, thereby improving data retrieval efficiency and facilitating the generation of retrieval results more efficiently.

[0157] In this optional embodiment, the method may also include the following operations:

[0158] Obtain the multimodal dataset to be stored, which includes multiple sets of multimodal data;

[0159] Based on the data compression interface corresponding to the multimodal retrieval system, a compression operation is performed on the multimodal dataset to obtain a set of compressed files corresponding to the multimodal dataset. The set of compressed files includes multiple compressed files.

[0160] Extract the compressed file queue from the compressed file set;

[0161] Determine whether the compressed file queue meets the preset data processing conditions;

[0162] When it is determined that the compressed file queue meets the preset data processing conditions, a batch processing operation is performed on the compressed file queue based on the inference interface corresponding to the multimodal retrieval system to obtain the vector dataset to be stored corresponding to the target vector dimension.

[0163] Store the vector dataset to be stored in the vector database.

[0164] The multimodal dataset can include various unstructured data, such as image data, video data, text data, audio data, point cloud data, etc., and this embodiment of the invention is not limited thereto. For example, the multimodal dataset can include datasets such as ImageNet, COCO, and YouTube-8M, and this embodiment of the invention is not limited thereto.

[0165] As can be seen, this optional embodiment can also obtain a set of compressed files after compressing the multimodal data, extract a queue of compressed files from the set of compressed files, and perform batch processing on the compressed files when the queue of compressed files meets the preset data processing conditions to obtain a vector dataset to be stored and store it in the vector database. This can efficiently store multimodal data in the vector database, improve the processing efficiency and accuracy of the data to be stored, thereby improving the data storage efficiency and accuracy of the vector database, and thus helping to improve the reliability of the data stored in the vector data.

[0166] In this optional embodiment, further optionally, extracting the compressed file queue from the compressed file set may include the following operations:

[0167] Store the compressed file collection in an intermediate data storage space;

[0168] When it is determined that the number of compressed files stored in the intermediate data storage space is greater than the preset number of files, the compressed files corresponding to the preset number of files are extracted from the intermediate data storage space as a compressed file queue.

[0169] This improves the stability and accuracy of the compressed file queue, thereby increasing the processing efficiency of subsequent compressed file queues.

[0170] In this optional embodiment, further optionally, determining whether the compressed file queue meets the preset data processing conditions may include the following operations:

[0171] Get the current processing time of the compressed file corresponding to the multimodal retrieval system;

[0172] Determine whether the queue length of the compressed file queue is greater than or equal to the preset number of files, whether the context of the inference model corresponding to the multimodal retrieval system has been created, and whether the current processing time of the compressed file exceeds the preset time.

[0173] When it is determined that the queue length of the compressed file queue is greater than or equal to the preset number of files, the context creation of the inference model corresponding to the multimodal retrieval system is completed, and the current processing time of the compressed file does not exceed the preset time, the compressed file queue is determined to meet the preset data processing conditions.

[0174] If it is determined that the queue length of the compressed file queue is less than the preset number of files, or the context of the inference model corresponding to the multimodal retrieval system has not been created, or the current processing time of the compressed file exceeds the preset time, then the compressed file queue does not meet the preset data processing conditions.

[0175] Optionally, the method may also include the following operations:

[0176] When it is determined that the context of the inference model corresponding to the multimodal retrieval system has not been fully created, resource initialization processing is performed on the inference model.

[0177] Optionally, the method may also include the following operations:

[0178] When all compressed files in the compressed file queue have been batch-processed and the current processing time of the compressed files exceeds the preset time, the model context is released.

[0179] As can be seen, this optional embodiment can also determine whether the compressed file queue meets the preset data processing conditions based on the queue length of the compressed file queue, the model context creation status, and the compressed file processing time. This can improve the accuracy of the condition determination of the compressed file queue, thereby improving the execution accuracy of subsequent batch processing operations and thus helping to improve data processing efficiency.

[0180] In this embodiment of the invention, an exemplary flowchart of the data retrieval method can be shown as follows: Figure 5 As shown, when the flow of this method is as follows Figure 5 As shown, specifically assuming the data to be retrieved includes video data A, the method may include the following operations:

[0181] Keyframe B can be extracted from video data A, and image C can be obtained after preprocessing keyframe B. Image C can then be vectorized (i.e.,...). Figure 5 The embedding of image C is used to obtain the vector corresponding to image C. The above steps are repeated until all keyframes of video A are vectorized to obtain the feature vector corresponding to video A. The feature vector is then converted into a vector index corresponding to the IVF-PQ index structure. Finally, based on this vector index, the retrieval of video A is completed in the FAISS vector data.

[0182] Example 2

[0183] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data retrieval method disclosed in an embodiment of the present invention. Figure 2The described data retrieval method can be applied to a data retrieval device, which may include one of a retrieval equipment, a retrieval terminal, and a server. The server may include a local server or a cloud server; this embodiment of the invention is not limited thereto. The device can be applied to a multimodal retrieval system, which includes a vector database and a pre-trained feature extraction model. The method can also be applied to a private cloud storage system; optionally, the private cloud storage system may be used with NAS services; this embodiment of the invention is not limited thereto. Figure 2 As shown, the data retrieval method may include the following operations:

[0184] 201. Obtain the data to be retrieved.

[0185] In this embodiment of the invention, the data to be retrieved includes at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data, and video data.

[0186] In this embodiment of the invention, optionally, before performing step 202, the method may further include the following operations:

[0187] Determine if the context corresponding to the feature extraction model is available;

[0188] When it is determined that the context corresponding to the feature extraction model is available, proceed to step 202;

[0189] When it is determined that the context corresponding to the feature extraction model is unavailable, a context corresponding to the feature extraction model is created.

[0190] 202. Determine the feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed data to be retrieved from the feature extraction model.

[0191] In this embodiment of the invention, the feature extraction model includes at least one feature extraction sub-model, and each feature extraction sub-model has a mapping relationship with at least one data type. The mapping relationship can be used to select the feature extraction sub-model that is suitable for the data type, and this embodiment of the invention does not limit it.

[0192] 203. For each type of sub-data to be retrieved, based on the feature extraction sub-model, perform feature extraction operation on the sub-data to be retrieved to obtain the target feature vector corresponding to the sub-data to be retrieved.

[0193] 204. Based on the target feature vectors corresponding to all the sub-data to be retrieved, determine the fusion feature vector corresponding to the data to be retrieved.

[0194] 205. Based on the target index structure corresponding to the vector database, determine the vector index corresponding to the fused feature vector.

[0195] 206. Based on the determined retrieval algorithm, perform a retrieval operation on the vector index in the vector database to obtain the retrieval results corresponding to the data to be retrieved.

[0196] In this embodiment of the invention, for other detailed descriptions of steps 201 and 203-204, please refer to the detailed descriptions of steps 101 and 103-104 in Embodiment 1. These descriptions will not be repeated in this embodiment of the invention.

[0197] As can be seen, the method described in the embodiments of the present invention can, after acquiring the data to be retrieved, perform feature extraction and fusion on the preprocessed data to be retrieved based on the feature extraction model to obtain a fused feature vector. Based on the target index structure corresponding to the vector database, the vector index corresponding to the fused feature vector is determined. Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain the retrieval results. This method can realize vector retrieval for multimodal data based on the vector database and the feature extraction model, thereby achieving multimodal data retrieval based on data semantic content. This improves the flexibility and accuracy of unstructured data analysis, thereby improving the flexibility and accuracy of multimodal data analysis, and thus improving the flexibility and accuracy of data retrieval. In turn, it generates accurate and reliable retrieval results, which is conducive to providing users with more efficient retrieval services. Furthermore, based on the feature extraction sub-model corresponding to each type of sub-data to be retrieved, feature extraction operations can be performed on the sub-data to be retrieved to obtain the corresponding target feature vectors. Then, based on all the target feature vectors, the fusion feature vector corresponding to the data to be retrieved can be determined. This can improve the flexibility and accuracy of feature extraction methods, thereby improving the accuracy and flexibility of feature extraction of the data to be retrieved, and further improving the accuracy of the extracted feature vectors. This is beneficial to improving the accuracy of the fusion feature vectors obtained by multimodal feature fusion, which in turn helps to further improve the accuracy of data retrieval.

[0198] In an optional embodiment, determining the feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed data to be retrieved from the feature extraction model may include the following operations:

[0199] For each type of sub-data to be retrieved, when the sub-data to be retrieved is image data or video data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined to be an image feature extraction sub-model, and the first feature extraction sub-model adopts a self-attention mechanism;

[0200] For each type of sub-data to be retrieved, when the sub-data to be retrieved is text data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined as the second feature extraction sub-model;

[0201] Furthermore, for each type of sub-data to be retrieved, based on the feature extraction sub-model, a feature extraction operation is performed on the sub-data to be retrieved to obtain the target feature vector corresponding to the sub-data to be retrieved. This may include the following operations:

[0202] For each type of sub-data to be retrieved, when the sub-data to be retrieved is image data or video data, the semantic features corresponding to the sub-data to be retrieved are extracted based on the self-attention mechanism of the first feature extraction sub-model, and used as the target feature vector corresponding to the sub-data to be retrieved.

[0203] For each type of sub-data to be retrieved, when the sub-data to be retrieved is text data, the sentence-level features corresponding to the sub-data to be retrieved are extracted based on the second feature extraction sub-model and used as the target feature vector corresponding to the sub-data to be retrieved.

[0204] Sentence-level features are used to represent the feature vector corresponding to a sentence in the data.

[0205] Self-attention mechanisms are used to focus more on key regions in image or video data during data processing, thereby better handling long-distance dependencies and capturing global information. Specifically, self-attention mechanisms model the global context by calculating the attention weights between each element and other elements in the data; the calculation formula for self-attention is shown below:

[0206]

[0207] Where Q represents query, K represents index, and V represents content, d k The index and content dimensions are represented by softmax, which is a normalization operation for similarity.

[0208] For example, the first feature extraction sub-model can be a convolutional neural network model improved from the model structure based on the Transformer model; the second feature extraction sub-model can be the SBERT (Sentence-BERT) model, which is not limited in this embodiment of the invention; wherein, the SBERT model is based on the BERT (Bidirectional Encoder Representation from Transformers) model, which is suitable for processing sentence-level tasks.

[0209] As can be seen, this optional embodiment can determine the model corresponding to the sub-data to be retrieved as the first feature extraction sub-model when the sub-data to be retrieved is image data or video data, and extract the semantic features corresponding to the sub-data to be retrieved as the target feature vector; when the sub-data to be retrieved is text data, it can determine the model corresponding to the sub-data to be retrieved as the second feature extraction sub-model, and extract the sentence-level features corresponding to the sub-data to be retrieved as the target feature vector. This can further improve the flexibility of feature extraction model selection, thereby increasing the degree of fit between the selected feature model and the data to be retrieved, and thus helping to improve the accuracy of the extracted feature vector.

[0210] In another optional embodiment, determining the fused feature vector corresponding to the data to be retrieved based on the target feature vectors corresponding to all the sub-data to be retrieved may include the following operations:

[0211] When the data to be retrieved includes only one type of sub-data to be retrieved, the target feature vector corresponding to the sub-data to be retrieved is determined as the fusion feature vector corresponding to the data to be retrieved.

[0212] When the data to be retrieved includes at least two types of sub-data to be retrieved, based on the multimodal fusion strategy corresponding to the multimodal retrieval system, feature fusion processing is performed on the target feature vectors corresponding to all sub-data to be retrieved to obtain the fused feature vector corresponding to the data to be retrieved.

[0213] Optionally, the multimodal fusion strategy may include a joint representation strategy or a collaborative representation strategy, which is not limited in this embodiment of the invention; wherein, the joint representation strategy is used to map all target feature vectors to a vector space to fuse multimodal vector features.

[0214] As can be seen, this optional embodiment can directly determine the target feature vector as the fused feature vector corresponding to the data to be retrieved when there is only one type of data to be retrieved. If there are multiple types of data to be retrieved, feature fusion processing is performed on all target feature vectors based on the multimodal fusion strategy to obtain the fused feature vector. This can improve the flexibility and accuracy of determining the fused feature vector, thereby improving the accuracy of the fused feature vector. This is beneficial for completing complex multimodal data retrieval and thus improving the accuracy of data retrieval.

[0215] In this embodiment of the invention, the flowchart of the data retrieval method can also be illustrated as follows: Figure 6 As shown, when the flow of this method is as follows Figure 6 As shown, specifically, the method may include a data retrieval process and a data storage process, wherein the multimodal data retrieval process may include the following operations:

[0216] Data to be retrieved is uploaded via text / image / audio upload interfaces. After the data upload is complete, the data to be retrieved is input into a compression interface to obtain compressed data to be retrieved. After all the data to be retrieved is compressed, the availability of the context (Infer) corresponding to the multimodal retrieval system is checked. If it is not available, a context is created. If it is available, features are extracted from the data to be retrieved to obtain a fused feature vector. After feature extraction, the model context is released. Multimodal retrieval is then performed on the vector database based on the fused feature vector to obtain the retrieval results.

[0217] The data storage process may include the following operations:

[0218] Upload the multimodal dataset to be stored via the image / audio upload interface. After the data upload is complete, input the multimodal dataset into the file compression interface to obtain the compressed file set. After compression, store the compressed file set in the intermediate data storage space. Figure 6 The psql (a type of database) shown above, when it is determined that the number of compressed files in the intermediate data storage space is greater than the preset number of files, fills the compressed file queue with compressed files. If the current compressed file queue is full or a timeout occurs, it checks whether the context (Infer) of the inference model is empty. If it is empty, it creates the context. If it is not empty, it directly reads the compressed files and inputs them into the model inference interface (i.e., the inference interface) to perform batch processing on the compressed files. The vector dataset to be stored after batch processing is stored in the vector database. If all compressed files have been processed and the timeout occurs and the compression queue is empty, the context of the inference model is released.

[0219] Example 3

[0220] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a data retrieval device disclosed in an embodiment of the present invention. Wherein, Figure 7 The described data retrieval apparatus may include one of a retrieval device, a retrieval terminal, and a server, wherein the server may include a local server or a cloud server, and this embodiment of the invention is not limited thereto; the apparatus may be applied to a multimodal retrieval system, which includes a vector database and a pre-trained feature extraction model; the apparatus may also be applied to a private cloud storage system, optionally, the private cloud storage system may be applied to NAS services, and this embodiment of the invention is not limited thereto. Figure 7 As shown, the data retrieval device may include:

[0221] The acquisition module 301 is used to acquire data to be retrieved, which includes at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data, and video data.

[0222] The feature extraction and fusion module 302 is used to perform feature extraction and fusion operations on the preprocessed data to be retrieved based on the feature extraction model, so as to obtain the fused feature vector corresponding to the data to be retrieved.

[0223] The determination module 303 is used to determine the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database;

[0224] The retrieval module 304 is used to perform a retrieval operation on the vector index in the vector database based on the determined retrieval algorithm, and obtain the retrieval results corresponding to the data to be retrieved.

[0225] As can be seen, the apparatus described in the embodiments of the present invention can, after acquiring the data to be retrieved, perform feature extraction and fusion on the preprocessed data to be retrieved based on a feature extraction model to obtain a fused feature vector. Based on the target index structure corresponding to the vector database, it determines the vector index corresponding to the fused feature vector. Based on the determined retrieval algorithm, it performs a retrieval operation on the vector index in the vector database to obtain retrieval results. It can realize vector retrieval for multimodal data based on the vector database and the feature extraction model, thereby realizing multimodal data retrieval based on data semantic content. This improves the flexibility and accuracy of unstructured data analysis, thereby improving the flexibility and accuracy of multimodal data analysis, and thus improving the flexibility and accuracy of data retrieval. In turn, it generates accurate and reliable retrieval results, which is conducive to providing users with more efficient retrieval services.

[0226] In an optional embodiment, the feature extraction model includes at least one feature extraction sub-model, and each feature extraction sub-model has a mapping relationship with at least one data type;

[0227] The feature extraction and fusion module 302, based on a feature extraction model, performs feature extraction and fusion operations on the preprocessed data to be retrieved, and the specific methods for obtaining the fused feature vector corresponding to the data to be retrieved may include:

[0228] The feature extraction model is used to determine the feature extraction sub-model corresponding to each type of sub-data in the preprocessed data to be retrieved;

[0229] For each type of sub-data to be retrieved, based on the feature extraction sub-model, a feature extraction operation is performed on the sub-data to be retrieved to obtain the target feature vector corresponding to the sub-data to be retrieved.

[0230] Based on the target feature vectors corresponding to all the sub-data to be retrieved, determine the fused feature vector corresponding to the data to be retrieved.

[0231] As can be seen, the apparatus described in this optional embodiment can perform feature extraction operations on the sub-data to be retrieved based on the feature extraction sub-model corresponding to each determined sub-data to be retrieved, obtain the corresponding target feature vector, and then determine the fusion feature vector corresponding to the data to be retrieved based on all target feature vectors. This can improve the flexibility and accuracy of feature extraction method selection, thereby improving the accuracy and flexibility of feature extraction of the data to be retrieved, and further improving the accuracy of the extracted feature vector. This is beneficial to improving the accuracy of the fusion feature vector obtained by multimodal feature fusion, which is conducive to further improving the accuracy of data retrieval.

[0232] In this optional embodiment, the specific method by which the feature extraction fusion module 302 determines the feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed retrieval data from the feature extraction model may include:

[0233] For each type of sub-data to be retrieved, when the sub-data to be retrieved is image data or video data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined to be an image feature extraction sub-model, and the first feature extraction sub-model adopts a self-attention mechanism;

[0234] For each type of sub-data to be retrieved, when the sub-data to be retrieved is text data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined as the second feature extraction sub-model;

[0235] Furthermore, the feature extraction and fusion module 302 performs feature extraction operations on each type of sub-data to be retrieved based on the feature extraction sub-model to obtain the target feature vector corresponding to the sub-data to be retrieved. The specific methods may include:

[0236] For each type of sub-data to be retrieved, when the sub-data to be retrieved is image data or video data, the semantic features corresponding to the sub-data to be retrieved are extracted based on the self-attention mechanism of the first feature extraction sub-model, and used as the target feature vector corresponding to the sub-data to be retrieved.

[0237] For each type of sub-data to be retrieved, when the sub-data to be retrieved is text data, the sentence-level features corresponding to the sub-data to be retrieved are extracted based on the second feature extraction sub-model and used as the target feature vector corresponding to the sub-data to be retrieved.

[0238] As can be seen, the apparatus described in this optional embodiment can also determine the model corresponding to the sub-data to be retrieved as a first feature extraction sub-model when the sub-data to be retrieved is image data or video data, and extract the semantic features corresponding to the sub-data to be retrieved as the target feature vector; when the sub-data to be retrieved is text data, it can determine the model corresponding to the sub-data to be retrieved as a second feature extraction sub-model, and extract the sentence-level features corresponding to the sub-data to be retrieved as the target feature vector. This can further improve the flexibility of feature extraction model selection, thereby increasing the degree of fit between the selected feature model and the data to be retrieved, and thus helping to improve the accuracy of the extracted feature vector.

[0239] In this optional embodiment, the specific method by which the feature extraction and fusion module 302 determines the fused feature vector corresponding to the data to be retrieved based on the target feature vectors corresponding to all the sub-data to be retrieved may include:

[0240] When the data to be retrieved includes only one type of sub-data to be retrieved, the target feature vector corresponding to the sub-data to be retrieved is determined as the fusion feature vector corresponding to the data to be retrieved.

[0241] When the data to be retrieved includes at least two types of sub-data to be retrieved, based on the multimodal fusion strategy corresponding to the multimodal retrieval system, feature fusion processing is performed on the target feature vectors corresponding to all sub-data to be retrieved to obtain the fused feature vector corresponding to the data to be retrieved.

[0242] As can be seen, the apparatus described in this optional embodiment can also directly determine the target feature vector as the fused feature vector corresponding to the data to be retrieved when there is only one type of data to be retrieved. If there are multiple types of data to be retrieved, feature fusion processing is performed on all target feature vectors based on a multimodal fusion strategy to obtain a fused feature vector. This can improve the flexibility and accuracy of determining the fused feature vector, thereby improving the accuracy of the fused feature vector. This is beneficial for completing complex multimodal data retrieval and thus improving the accuracy of data retrieval.

[0243] In another optional embodiment, the acquisition module 301 is further configured to acquire the business scenario corresponding to the data to be retrieved and the retrieval requirements corresponding to the data to be retrieved;

[0244] Furthermore, the specific method by which the retrieval module 304 performs a retrieval operation on the vector index in the vector database based on the determined retrieval algorithm to obtain the retrieval results corresponding to the data to be retrieved may include:

[0245] Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain multiple candidate retrieval results;

[0246] Analyze the business relevance of each candidate search result to the business scenario;

[0247] Filter out at least one target relevance that is greater than or equal to the preset business relevance from all business relevance scores;

[0248] Based on the candidate search results corresponding to all target relevance, the candidate search results that meet the search requirements are determined as the search results corresponding to the data to be searched.

[0249] As can be seen, the apparatus described in this optional embodiment can perform a retrieval operation on the vector index in the vector database based on a determined retrieval algorithm to obtain multiple candidate retrieval results, analyze the business relevance between each candidate retrieval result and the business scenario, and then filter at least one target relevance greater than or equal to a preset business relevance from all business relevances. Based on the candidate retrieval results corresponding to the target relevance, the apparatus determines the candidate retrieval results that meet the retrieval requirements as the retrieval results. This can improve the accuracy of the association analysis of candidate retrieval results with respect to the business scenario, thereby improving the flexibility and accuracy of retrieval result filtering. In turn, it helps to make the filtered retrieval results more in line with the retrieval requirements, and further improves the accuracy and reliability of the retrieval results.

[0250] In another optional embodiment, the acquisition module 301 is further configured to acquire the test vector dimension corresponding to the vector database and the test vector dataset corresponding to the test vector dimension;

[0251] The retrieval module 302 is also used to perform a retrieval operation on the test vector dataset in the vector database based on the test vector dataset corresponding to the test vector dimension, and obtain the test retrieval results corresponding to the test vector dimension;

[0252] Among them, such as Figure 8 As shown, the device may further include:

[0253] Analysis module 305 is used to analyze the test retrieval results and obtain the retrieval performance corresponding to the test vector dimension;

[0254] The adjustment module 306 is used to adjust the test vector dimension based on the predetermined retrieval performance standard and retrieval performance to obtain the target vector dimension corresponding to the vector database.

[0255] Furthermore, the specific method by which module 303 determines the vector index corresponding to the fused feature vector based on the target index structure of the vector database may include:

[0256] Based on the target index structure corresponding to the vector database, the fused feature vector is transformed into a vector index corresponding to the target vector dimension.

[0257] As can be seen, the apparatus described in this optional embodiment can test the data retrieval performance of the test vector dimension based on the test vector dataset corresponding to the test vector dimension. Based on the retrieval performance standard, the test vector dimension can be adjusted to obtain the target vector dimension. This improves the accuracy of determining the vector dimension corresponding to the vector database, thereby increasing the precision of the vector dimension. This helps to reduce vector storage space while ensuring sufficient information in the vector storage of the database, and improves data retrieval efficiency, facilitating more efficient generation of retrieval results. Furthermore, based on the target index result and the target vector dimension, the fused feature vector can be transformed into a vector index, improving the efficiency of data matching in the vector database based on the vector index, thus improving data retrieval efficiency and facilitating more efficient generation of retrieval results.

[0258] In this optional embodiment, the acquisition module 301 is optionally further configured to acquire a multimodal dataset to be stored, the multimodal dataset including multiple sets of multimodal data;

[0259] Among them, such as Figure 8 As shown, the device may further include:

[0260] Compression module 307 is used to perform compression operations on the multimodal dataset based on the data compression interface corresponding to the multimodal retrieval system, and obtain a set of compressed files corresponding to the multimodal dataset. The set of compressed files includes multiple compressed files.

[0261] Extraction module 308 is used to extract a queue of compressed files from a set of compressed files;

[0262] The judgment module 309 is used to determine whether the compressed file queue meets the preset data processing conditions;

[0263] The batch processing module 310 is used to perform batch processing operations on the compressed file queue based on the inference interface corresponding to the multimodal retrieval system when the judgment module 309 determines that the compressed file queue meets the preset data processing conditions, so as to obtain the vector dataset to be stored corresponding to the target vector dimension.

[0264] Storage module 311 is used to store the vector dataset to be stored into the vector database.

[0265] As can be seen, the apparatus described in this optional embodiment can also obtain a set of compressed files after compressing multimodal data, extract a queue of compressed files from the set of compressed files, and perform batch processing on the compressed files when the queue of compressed files meets preset data processing conditions to obtain a vector dataset to be stored and store it in a vector database. This can efficiently store multimodal data in a vector database, improve the processing efficiency and accuracy of the data to be stored, thereby improving the data storage efficiency and accuracy of the vector database, and thus helping to improve the reliability of the data stored in the vector data.

[0266] Example 4

[0267] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of another network-attached storage device disclosed in an embodiment of the present invention. For example... Figure 9 As shown, the network-attached storage device may include:

[0268] Memory 401 storing executable program code;

[0269] Processor 402 coupled to memory 401;

[0270] The processor 402 calls the executable program code stored in the memory 401 to execute the steps of the data retrieval method described in Embodiment 1 or Embodiment 2 of the present invention.

[0271] Example 5

[0272] This invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute steps in the data retrieval method described in Embodiment 1 or Embodiment 2 of this invention.

[0273] Example 6

[0274] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of the data retrieval method described in Embodiment 1 or Embodiment 2.

[0275] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0276] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0277] Finally, it should be noted that the data retrieval method and apparatus and network attached storage device disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data retrieval method, characterized in that, The method is applied to a multimodal retrieval system, which includes a vector database and a pre-trained feature extraction model. The method includes: Acquire data to be retrieved, the data to be retrieved including at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data, and video data; Based on the feature extraction model, a feature extraction and fusion operation is performed on the preprocessed data to be retrieved to obtain the fused feature vector corresponding to the data to be retrieved. Based on the target index structure corresponding to the vector database, the vector index corresponding to the fused feature vector is determined; the target index structure is a combined index structure composed of HNSW graph structure and PQ. Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain the retrieval results corresponding to the data to be retrieved; The method further includes: Obtain the business scenario corresponding to the data to be retrieved and the retrieval requirements corresponding to the data to be retrieved; wherein, the business scenario belongs to one or more of the following business scenario types: recommendation type, image type, video type, audio type, NLP type, spatiotemporal type, and interdisciplinary type; the retrieval requirements include one or more of the following combinations: the number of retrieval results, the preset correlation between the retrieval results and the data to be retrieved, retrieval time, and retrieval memory consumption. And, based on the determined retrieval algorithm, performing a retrieval operation on the vector index in the vector database to obtain the retrieval results corresponding to the data to be retrieved includes: Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain multiple candidate retrieval results; Analyze the business relevance of each candidate search result to the business scenario; Filter out at least one target correlation degree that is greater than or equal to the preset business correlation degree from all the said business correlation degrees; Based on all the candidate search results corresponding to the target relevance, the candidate search results that meet the search requirements are determined as the search results corresponding to the data to be searched; The method further includes: The data to be retrieved is compressed to obtain the compressed data to be retrieved. For each type of sub-data to be retrieved in the compressed data to be retrieved, a preprocessing operation corresponding to the data type of the sub-data to be retrieved is performed to obtain the preprocessed sub-data to be retrieved; wherein, the preprocessed data to be retrieved includes at least one of the image data after image enhancement, the video data after frame extraction, and the text data after semantic enhancement; The vector database stores data in the following manner: Obtain the multimodal dataset to be stored, wherein the multimodal dataset includes multiple sets of multimodal data; Based on the data compression interface corresponding to the multimodal retrieval system, a compression operation is performed on the multimodal dataset to obtain a set of compressed files corresponding to the multimodal dataset, wherein the set of compressed files includes multiple compressed files; Extract the compressed file queue from the set of compressed files; Determine whether the compressed file queue meets the preset data processing conditions; When it is determined that the compressed file queue meets the preset data processing conditions, a batch processing operation is performed on the compressed file queue based on the inference interface corresponding to the multimodal retrieval system to obtain the vector dataset to be stored corresponding to the target vector dimension. Store the vector dataset to be stored into the vector database; The step of determining whether the compressed file queue meets the preset data processing conditions includes: Obtain the current processing time of the compressed file corresponding to the multimodal retrieval system; Determine whether the queue length of the compressed file queue is greater than or equal to the preset number of files, whether the context of the inference model corresponding to the multimodal retrieval system has been created, and whether the current processing time of the compressed file exceeds the preset time. When it is determined that the queue length of the compressed file queue is greater than or equal to the preset number of files, the context creation of the inference model corresponding to the multimodal retrieval system is completed, and the current processing time of the compressed file does not exceed the preset time, the compressed file queue is determined to meet the preset data processing conditions.

2. The data retrieval method according to claim 1, characterized in that, The feature extraction model includes at least one feature extraction sub-model, and each feature extraction sub-model has a mapping relationship with at least one of the data types; The step of performing feature extraction and fusion operations on the preprocessed data to be retrieved based on the feature extraction model to obtain the fused feature vector corresponding to the data to be retrieved includes: From the feature extraction model, determine the feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed data to be retrieved; For each type of sub-data to be retrieved, based on the feature extraction sub-model, a feature extraction operation is performed on the sub-data to be retrieved to obtain the target feature vector corresponding to the sub-data to be retrieved. Based on the target feature vectors corresponding to all the sub-data to be retrieved, determine the fused feature vector corresponding to the data to be retrieved.

3. The data retrieval method according to claim 2, characterized in that, The step of determining the feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed data to be retrieved from the feature extraction model includes: For each type of sub-data to be retrieved, when the sub-data to be retrieved is the image data or the video data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined to be the first feature extraction sub-model, and the first feature extraction model adopts a self-attention mechanism; For each type of sub-data to be retrieved, when the sub-data to be retrieved is the text data, the feature extraction sub-model corresponding to the sub-data to be retrieved is determined as the second feature extraction sub-model; And, for each type of sub-data to be retrieved, based on the feature extraction sub-model, performing a feature extraction operation on the sub-data to be retrieved to obtain the target feature vector corresponding to the sub-data to be retrieved includes: For each type of sub-data to be retrieved, when the sub-data to be retrieved is the image data or the video data, the semantic features corresponding to the sub-data to be retrieved are extracted based on the self-attention mechanism of the first feature extraction sub-model, and used as the target feature vector corresponding to the sub-data to be retrieved. For each type of sub-data to be retrieved, when the sub-data to be retrieved is text data, the sentence-level features corresponding to the sub-data to be retrieved are extracted based on the second feature extraction sub-model and used as the target feature vector corresponding to the sub-data to be retrieved.

4. The data retrieval method according to claim 2, characterized in that, The step of determining the fused feature vector corresponding to the data to be retrieved based on the target feature vectors corresponding to all the sub-data to be retrieved includes: When the data to be retrieved includes only one type of sub-data to be retrieved, the target feature vector corresponding to the sub-data to be retrieved is determined as the fusion feature vector corresponding to the data to be retrieved. When the data to be retrieved includes at least two types of sub-data to be retrieved, based on the multimodal fusion strategy corresponding to the multimodal retrieval system, feature fusion processing is performed on the target feature vectors corresponding to all the sub-data to be retrieved to obtain the fused feature vector corresponding to the data to be retrieved.

5. The data retrieval method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the test vector dimension corresponding to the vector database and the test vector dataset corresponding to the test vector dimension; Based on the test vector dataset corresponding to the test vector dimension, a retrieval operation on the test vector dataset is performed in the vector database to obtain the test retrieval results corresponding to the test vector dimension; Analyze the test retrieval results to obtain the retrieval performance corresponding to the test vector dimension; Based on the predetermined retrieval performance standard and the retrieval performance, the test vector dimension is adjusted to obtain the target vector dimension corresponding to the vector database; And, determining the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database includes: Based on the target index structure corresponding to the vector database, the fused feature vector is transformed into a vector index corresponding to the target vector dimension.

6. A data retrieval device, characterized in that, The apparatus is used to perform the data retrieval method as described in any one of claims 1-5, and the apparatus is applied to a multimodal retrieval system, the multimodal retrieval system comprising a vector database and a pre-trained feature extraction model, the apparatus comprising: The acquisition module is used to acquire data to be retrieved, the data to be retrieved including at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data and video data. The feature extraction and fusion module is used to perform feature extraction and fusion operations on the preprocessed data to be retrieved based on the feature extraction model, so as to obtain the fused feature vector corresponding to the data to be retrieved. The determining module is used to determine the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database; The retrieval module is used to perform a retrieval operation on the vector index in the vector database based on a determined retrieval algorithm, and obtain the retrieval results corresponding to the data to be retrieved.

7. A network-attached storage device, characterized in that, The network-attached storage device includes: Memory containing executable program code; A processor coupled to the memory; The processor invokes the executable program code stored in the memory to execute the data retrieval method as described in any one of claims 1-5.

8. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the data retrieval method as described in any one of claims 1-5.

Images