Data retrieval method and network-attached storage device

Abstract

The present invention relates to the technical field of data retrieval. Disclosed are a data retrieval method and a network-attached storage device. The method comprises: acquiring data to be retrieved, wherein said data comprises at least one piece of sub-data to be retrieved, and each piece of said sub-data comprises one of text data, image data, and video data; on the basis of a feature extraction model, performing a feature extraction fusion operation on the preprocessed data to be retrieved to obtain a fusion feature vector corresponding to the data to be retrieved; on the basis of a target index structure corresponding to a vector database, determining a vector index corresponding to the fusion feature vector; and on the basis of a determined retrieval algorithm, performing a retrieval operation on the vector index in the vector database to obtain a retrieval result corresponding to the data to be retrieved. Therefore, by implementing the present invention, the data retrieval flexibility can be improved, so that more efficient retrieval services can be provided for users.

Description

Data retrieval method and network attached storage device Technical Field

[0001] The present invention relates to the technical field of data retrieval, and in particular to a data retrieval method and a network attached storage device. Background Art

[0002] Since most users have the need to retrieve data when browsing websites or using applications, data retrieval has gradually become a major function of current websites and software, allowing users to easily obtain the information they need through retrieval.

[0003] In practice, it has been found that most existing search functions only implement data retrieval through traditional text retrieval methods. However, traditional retrieval methods are difficult to handle retrieval requirements for unstructured data (such as text, images, audio, etc.). Therefore, it is particularly important to propose a technical solution that can improve the flexibility of data retrieval and provide users with more efficient retrieval services.

[0004] Summary of the Invention

[0005] The present invention provides a data retrieval method and a network attached storage device, which can improve the flexibility of data retrieval and facilitate providing more efficient retrieval services to users.

[0006] In order to solve the above technical problems, the first aspect of the present invention discloses a data retrieval method, which is applied to a multimodal retrieval system. The multimodal retrieval system includes a vector database and a pre-trained feature extraction model. The method includes:

[0007] Acquire data to be retrieved, wherein the data to be retrieved includes at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data, and video data;

[0008] Based on the feature extraction model, performing a feature extraction fusion operation on the preprocessed data to be retrieved to obtain a fused feature vector corresponding to the data to be retrieved;

[0009] Determining a vector index corresponding to the fused feature vector based on a target index structure corresponding to the vector database;

[0010] Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain a retrieval result corresponding to the data to be retrieved.

[0011] A second aspect of the present invention discloses a network attached storage device, the network attached storage device comprising:

[0012] a memory storing executable program code;

[0013] a processor coupled to the memory;

[0014] The processor calls the executable program code stored in the memory to execute the data retrieval method disclosed in the first aspect of the present invention.

[0015] A third aspect of the present invention discloses a computer storage medium, wherein the computer storage medium stores computer instructions. When the computer instructions are called, they are used to execute the data retrieval method disclosed in the first aspect of the present invention.

[0016] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0017] The implementation of the present invention can, after obtaining the data to be retrieved, perform feature extraction and fusion on the pre-processed data to be retrieved based on the feature extraction model to obtain a fused feature vector, and determine the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database, and perform a retrieval operation on the vector index in the vector database based on the determined retrieval algorithm to obtain a retrieval result. It can realize vector retrieval for multimodal data based on the vector database and the feature extraction model, so as to realize multimodal data retrieval based on the semantic content of the data, improve the analysis flexibility and analysis accuracy of unstructured data, improve the analysis flexibility and analysis accuracy of multimodal data, thereby improving data retrieval flexibility and data retrieval accuracy, and then generate accurate and reliable retrieval results, which is conducive to providing users with more efficient retrieval services. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] FIG1 is a flow chart of a data retrieval method disclosed in an embodiment of the present invention;

[0020] FIG2 is a flow chart of another data retrieval method disclosed in an embodiment of the present invention;

[0021] FIG3 is a schematic diagram of the structure of an index structure disclosed in an embodiment of the present invention;

[0022] FIG4 is a schematic diagram of scenario types of business scenarios applicable to a data retrieval method disclosed in an embodiment of the present invention;

[0023] FIG5 is a flow chart of another data retrieval method disclosed in an embodiment of the present invention;

[0024] FIG6 is a flow chart of another data retrieval method disclosed in an embodiment of the present invention;

[0025] FIG7 is a schematic diagram of the structure of a network attached storage device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] for

[0027] The present invention discloses a data retrieval method and apparatus, and a network attached storage device. After obtaining the data to be retrieved, the method and apparatus can perform feature extraction and fusion on the pre-processed data to be retrieved based on a feature extraction model to obtain a fused feature vector, and based on the target index structure corresponding to the vector database, determine the vector index corresponding to the fused feature vector, and based on the determined retrieval algorithm, perform a retrieval operation on the vector index in the vector database to obtain a retrieval result. The method and apparatus can implement vector retrieval for multimodal data based on the vector database and the feature extraction model, so as to implement multimodal data retrieval based on the semantic content of the data, thereby improving the analysis flexibility and accuracy of unstructured data, improving the analysis flexibility and accuracy of multimodal data, thereby improving the data retrieval flexibility and accuracy, and then generating accurate and reliable retrieval results, which is conducive to providing users with more efficient retrieval services. The following are detailed descriptions.

[0028] Example 1

[0029] Please refer to Figure 1, which is a flow chart of a data retrieval method disclosed in an embodiment of the present invention. The data retrieval method described in Figure 1 can be applied to a data retrieval device, which may include a retrieval device, a retrieval terminal, and a server, wherein the server may include a local server or a cloud server, which is not limited by the embodiment of the present invention; the device can be applied to a multimodal retrieval system, which includes a vector database and a pre-trained feature extraction model; the method can also be applied to a private cloud storage system, and optionally, the private cloud storage system can be applied to a NAS service, which is not limited by the embodiment of the present invention. As shown in Figure 1, the data retrieval method may include the following operations:

[0030] 101. Get the data to be retrieved.

[0031] In an embodiment of the present invention, the data to be retrieved includes at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data and video data; optionally, the sub-data to be retrieved can also be other unstructured data, such as: audio data or point cloud data or user behavior data, which is not limited in the embodiment of the present invention.

[0032] 102. Based on the feature extraction model, perform a feature extraction fusion operation on the preprocessed data to be retrieved to obtain a fused feature vector corresponding to the data to be retrieved.

[0033] In the embodiment of the present invention, the feature extraction and fusion operation includes a feature extraction operation and a feature fusion operation.

[0034] 103. Based on the target index structure corresponding to the vector database, determine the vector index corresponding to the fused feature vector.

[0035] In the embodiment of the present invention, illustratively, the vector database in the embodiment of the present invention may be FAISS (Facebook AI Similarity Search, a vector database for clustering and similarity search open sourced by the Facebook AI team), or other vector databases, which is not limited in the embodiment of the present invention; wherein, the target index structure may be a pre-set index structure, optionally, the target index structure may be a combined index structure composed of an HNSW (Hierarchical Navigable Small World) graph structure and PQ (Product Quantization). By using this combined index structure, efficient approximate nearest neighbor search capability can be achieved by using HNSW, and by using PQ, the storage pressure of the database can be reduced while maintaining retrieval accuracy, thereby locating the most relevant retrieval structure in a large-scale data set and improving retrieval efficiency. The target index structure may also be other index structures, such as IVF (Inverted File Index), which is not limited in the embodiment of the present invention.

[0036] In an embodiment of the present invention, exemplarily, the index structure of the vector index can be as shown in Figure 3, which is a structural diagram of an index structure disclosed in an embodiment of the present invention. Specifically, the structure can be used to retrieve video data containing picture d, which is not limited in the embodiment of the present invention.

[0037] 104. Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain a retrieval result corresponding to the data to be retrieved.

[0038] In the embodiments of the present invention, the search algorithm may optionally be a deep learning-based query optimization algorithm or other search algorithm, which is not limited in the embodiments of the present invention. Optionally, there may be one or more search results whose correlation with the data to be searched is greater than a preset correlation, which is not limited in the embodiments of the present invention.

[0039] It can be seen that the method described in the embodiment of the present invention can, after obtaining the data to be retrieved, perform feature extraction and fusion on the pre-processed data to be retrieved based on the feature extraction model to obtain a fused feature vector, and determine the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database, and perform a retrieval operation on the vector index in the vector database based on the determined retrieval algorithm to obtain a retrieval result. It can realize vector retrieval for multimodal data based on the vector database and the feature extraction model, so as to realize multimodal data retrieval based on the semantic content of the data, improve the analysis flexibility and analysis accuracy of unstructured data, improve the analysis flexibility and analysis accuracy of multimodal data, thereby improving data retrieval flexibility and data retrieval accuracy, and then generate accurate and reliable retrieval results, which is conducive to providing users with more efficient retrieval services.

[0040] In an optional embodiment, the method may further include the following operations:

[0041] Performing compression processing on the data to be retrieved to obtain compressed data to be retrieved;

[0042] For each type of sub-data to be retrieved in the compressed data to be retrieved, a preprocessing operation corresponding to the data type of the sub-data to be retrieved is performed on the sub-data to be retrieved, so as to obtain preprocessed sub-data to be retrieved.

[0043] This can reduce the size of data required for subsequent data processing by compressing the data to be retrieved, which is beneficial to improving data analysis efficiency and reducing the memory required for subsequent data retrieval, thereby improving data retrieval efficiency; and, by preprocessing the data to be retrieved, the accuracy and efficiency of subsequent feature extraction can be improved.

[0044] In this optional embodiment, optionally, for each sub-data to be retrieved in the compressed data to be retrieved, performing a preprocessing operation corresponding to the data type of the sub-data to be retrieved on the sub-data to be retrieved to obtain the preprocessed sub-data to be retrieved may include the following operations:

[0045] For each type of sub-data to be retrieved in the compressed data to be retrieved, when the sub-data to be retrieved is image data, performing image quality enhancement processing on the image data to obtain enhanced image data;

[0046] When the sub-data to be retrieved is video data, the video data is decomposed into multiple video frames, and inter-frame correlation analysis is performed on all the video frames to obtain correlation analysis results; based on the correlation analysis results, at least one key frame is determined from all the video frames to obtain frame-extracted video data;

[0047] When the sub-data to be retrieved is text data, performing semantic enhancement processing on the text data to obtain semantically enhanced text data;

[0048] The pre-processed data to be retrieved includes at least one of image data after image enhancement, video data after frame extraction, and text data after semantic enhancement.

[0049] For example, the image data can be processed for image quality enhancement using GANs (Generative Adversarial Networks) technology; the text data can be processed for semantic enhancement using NLP (Natural Language Processing) technology, which is not limited in the embodiments of the present invention.

[0050] It can be seen that this optional embodiment can also select different preprocessing operations according to different data types, which can improve the flexibility and accuracy of data preprocessing, thereby improving the reliability of preprocessed data, and further helping to further improve the subsequent feature extraction accuracy and feature extraction efficiency.

[0051] In another optional embodiment, the method may further include the following operations:

[0052] Obtain the business scenarios and retrieval requirements corresponding to the data to be retrieved;

[0053] Furthermore, based on the determined search algorithm, performing a search operation on the vector index in the vector database to obtain a search result corresponding to the data to be searched may include the following operations:

[0054] Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain multiple candidate retrieval results;

[0055] Analyze the business relevance of each candidate search result to the business scenario;

[0056] Filtering at least one target relevance greater than or equal to a preset business relevance from all business relevances;

[0057] According to the candidate retrieval results corresponding to all target relevances, the candidate retrieval results that meet the retrieval requirements are determined as the retrieval results corresponding to the data to be retrieved.

[0058] Among them, the business scenario is used to represent the business that needs to be implemented based on the retrieval results corresponding to the data to be retrieved; among them, optionally, the retrieval requirements can be determined based on the business scenario, which is not limited in the embodiments of the present invention; the retrieval requirements may include a combination of one or more of the number of retrieval results, the preset correlation between the retrieval results and the data to be retrieved, the retrieval time and the retrieval memory consumption, which is not limited in the embodiments of the present invention.

[0059] Exemplarily, the business scenario corresponding to the data to be retrieved may be a combination of one or more of the business scenarios shown in Figure 4, which is not limited by the embodiment of the present invention; wherein, Figure 4 is a schematic diagram of the scenario types of business scenarios applicable to a data retrieval method disclosed in an embodiment of the present invention; specifically, the method can be applied to a combination of one or more business scenario types such as recommendation type, image type, video type, audio type, NLP type, spatiotemporal type and interdisciplinary type, which is not limited by the embodiment of the present invention.

[0060] It can be seen that this optional embodiment can, based on the determined retrieval algorithm, perform a retrieval operation on the vector index in the vector database to obtain multiple candidate retrieval results, and analyze the business correlation between each candidate retrieval result and the business scenario, and then screen out at least one target correlation greater than or equal to the preset business correlation from all business correlations, and based on the candidate retrieval results corresponding to the target correlation, determine the candidate retrieval results that meet the retrieval requirements as the retrieval results, which can improve the accuracy of the association analysis of the candidate retrieval results with respect to the business scenario, thereby improving the screening flexibility and screening accuracy of the retrieval results, and thus helping to make the screened retrieval results more in line with the retrieval requirements, and further improve the accuracy and reliability of the retrieval results.

[0061] In this optional embodiment, optionally, when the business scenario corresponding to the data to be retrieved is a private cloud storage scenario, the method may further include:

[0062] Determine the sensitive information corresponding to the private cloud storage scenario;

[0063] Determine whether there are any search results involving sensitive information among all candidate search results;

[0064] When it is determined that there is a search result involving sensitive information among all candidate search results, the search result involving sensitive information is determined as a filtered search result;

[0065] Eliminate all filtered search results from all candidate search results to obtain filtered candidate search results; and trigger an operation to analyze the business relevance of each candidate search result to the business scenario.

[0066] Sensitive information may include one or more types of information that needs to be protected, including user personal information, system security information, and account password information, which is not limited in the embodiment of the present invention.

[0067] It can be seen that this optional embodiment can also, when the business scenario is a private cloud storage scenario, determine whether the candidate retrieval results involve sensitive information, so as to eliminate the candidate retrieval results that are determined to involve sensitive information, and further screen the retrieval results, thereby further improving the screening accuracy of the retrieval results, thereby improving the information security provided to users by the retrieval results, and helping to improve the data security of the multimodal retrieval system.

[0068] In yet another optional embodiment, the method may further include the following operations:

[0069] Obtaining a test vector dimension corresponding to the vector database and a test vector dataset corresponding to the test vector dimension;

[0070] Based on the test vector data set corresponding to the test vector dimension, performing a search operation on the test vector data set in the vector database to obtain a test search result corresponding to the test vector dimension;

[0071] Analyze the test retrieval results and obtain the retrieval performance corresponding to the test vector dimension;

[0072] Based on the predetermined retrieval performance standard and retrieval performance, the test vector dimension is adjusted to obtain the target vector dimension corresponding to the vector database.

[0073] Among them, there may be one or more test vector dimensions; optionally, when there are multiple test vector dimensions, a test vector dimension with the best retrieval performance can be first selected from all test vector dimensions according to the retrieval performance as the test vector dimension to be adjusted, and then the test vector dimension can be adjusted based on the retrieval performance standard and the retrieval performance corresponding to the test vector dimension to be adjusted to obtain the target vector dimension corresponding to the vector database. This is not limited in the embodiments of the present invention.

[0074] Among them, the retrieval performance may include a combination of one or more of the retrieval time, retrieval speed and retrieval memory consumption, which is not limited in the embodiment of the present invention; illustratively, the target vector dimension can be 576 or other values, which is not limited in the embodiment of the present invention.

[0075] It can be seen that this optional embodiment can test the data retrieval performance of the test vector dimension based on the test vector data set corresponding to the test vector dimension, and thus adjust the test vector dimension based on the retrieval performance standard to obtain the target vector dimension, which can improve the accuracy of determining the vector dimension corresponding to the vector database, thereby improving the accuracy of the vector dimension corresponding to the vector database, and thus helping to reduce the vector storage space while ensuring that the vectors stored in the database have sufficient information storage, and improve data retrieval efficiency, which is conducive to more efficient generation of retrieval results.

[0076] In this optional embodiment, optionally, determining the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database may include the following operations:

[0077] Based on the target index structure corresponding to the vector database, the fused feature vector is converted into a vector index corresponding to the target vector dimension.

[0078] Exemplarily, the above-mentioned vector index may be a vector index having a structure composed of an HNSW graph structure and PQ as an index structure and 576 as a vector dimension, which is not limited in the embodiment of the present invention.

[0079] It can be seen that this optional embodiment can also convert the fused feature vector into a vector index based on the target index result and the target vector dimension, which can improve the efficiency of data matching in the vector database based on the vector index, thereby improving the data retrieval efficiency and facilitating more efficient generation of retrieval results.

[0080] In this optional embodiment, the method may further include the following operations:

[0081] Acquire a multimodal data set to be stored, where the multimodal data set includes multiple sets of multimodal data;

[0082] Based on a data compression interface corresponding to the multimodal retrieval system, a compression operation is performed on the multimodal dataset to obtain a compressed file set corresponding to the multimodal dataset, where the compressed file set includes multiple compressed files;

[0083] Extracting a compressed file queue from the compressed file collection;

[0084] Determine whether the compressed file queue meets the preset data processing conditions;

[0085] When it is determined that the compressed file queue meets the preset data processing conditions, a batch processing operation is performed on the compressed file queue based on the reasoning interface corresponding to the multimodal retrieval system to obtain a vector data set to be stored corresponding to the target vector dimension;

[0086] Store the vector dataset to be stored in the vector database.

[0087] The multimodal dataset may include a variety of unstructured data, such as image data, video data, text data, audio data, point cloud data, etc., which is not limited in the present embodiment. For example, the multimodal dataset may include ImageNet, COCO, YouTube-8M, etc., which is not limited in the present embodiment.

[0088] It can be seen that this optional embodiment can also obtain a compressed file set after compressing the multimodal data, and extract a compressed file queue from the compressed file set, and when the compressed file queue meets the preset data processing conditions, batch process the compressed files to obtain the vector data set to be stored and store it in the vector database, which can efficiently store the multimodal data in the vector database, improve the processing efficiency and accuracy of the data to be stored, thereby improving the data storage efficiency and data storage accuracy of the vector database, and thus helping to improve the reliability of the data stored in the vector data.

[0089] In this optional embodiment, further optionally, extracting the compressed file queue from the compressed file set may include the following operations:

[0090] Storing the compressed file collection in an intermediate data storage space;

[0091] When it is determined that the number of compressed files stored in the intermediate data storage space is greater than the preset number of files, compressed files corresponding to the preset number of files are extracted from the intermediate data storage space as a compressed file queue.

[0092] This can improve the stability and accuracy of determining the compressed file queue, thereby facilitating improved efficiency in subsequent processing of the compressed file queue.

[0093] In this optional embodiment, further optionally, determining whether the compressed file queue meets the preset data processing condition may include the following operations:

[0094] Get the current processing time of the compressed file corresponding to the multimodal retrieval system;

[0095] Determine whether the queue length of the compressed file queue is greater than or equal to the preset number of files, whether the context of the inference model corresponding to the multimodal retrieval system is created, and whether the current processing time of the compressed file exceeds the preset time;

[0096] When it is determined that the queue length of the compressed file queue is greater than or equal to the preset number of files, the context of the inference model corresponding to the multimodal retrieval system is created, and the current processing time of the compressed file does not exceed the preset time, it is determined that the compressed file queue meets the preset data processing condition;

[0097] When it is determined that the queue length of the compressed file queue is less than the preset number of files or the context of the reasoning model corresponding to the multimodal retrieval system has not been created or the current processing time of the compressed file exceeds the preset time, it is determined that the compressed file queue does not meet the preset data processing conditions.

[0098] Optionally, the method may further include the following operations:

[0099] When it is determined that the context of the inference model corresponding to the multimodal retrieval system has not been created, resource initialization processing is performed on the inference model.

[0100] Optionally, the method may further include the following operations:

[0101] When all compressed files in the compressed file queue are batch processed and the current compressed file processing time exceeds the preset time, the model context is released.

[0102] It can be seen that this optional embodiment can also determine whether the compressed file queue meets the preset data processing conditions based on the queue length of the compressed file queue, the model context creation status, and the compressed file processing time, which can improve the accuracy of the condition determination of the compressed file queue, thereby improving the execution accuracy of subsequent batch operations, and thus helping to improve data processing efficiency.

[0103] In an embodiment of the present invention, a flowchart of the data retrieval method may be shown in FIG5 . When the flowchart of the method is shown in FIG5 , specifically, assuming that the data to be retrieved includes video data A, the method may include the following operations:

[0104] Key frame B can be extracted from video data A, and key frame B is preprocessed to obtain image C. Image C is vectorized (i.e., embedding in Figure 5) to obtain the vector corresponding to image C. The above steps are repeated until all key frames of video A are vectorized to obtain the feature vector corresponding to video A. The feature vector is converted into a vector index corresponding to the index structure of IVF-PQ, and then the retrieval of video A is completed in the FAISS vector data based on the vector index.

[0105] Example 2

[0106] Please refer to Figure 2, which is a flow chart of a data retrieval method disclosed in an embodiment of the present invention. The data retrieval method described in Figure 2 can be applied to a data retrieval device, which may include a retrieval device, a retrieval terminal, and a server, wherein the server may include a local server or a cloud server, which is not limited by the embodiment of the present invention; the device can be applied to a multimodal retrieval system, which includes a vector database and a pre-trained feature extraction model; the method can also be applied to a private cloud storage system, and optionally, the private cloud storage system can be applied to a NAS service, which is not limited by the embodiment of the present invention. As shown in Figure 2, the data retrieval method may include the following operations:

[0107] 201. Get the data to be retrieved.

[0108] In the embodiment of the present invention, the data to be retrieved includes at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data and video data.

[0109] In the embodiment of the present invention, optionally, before executing step 202, the method may further include the following operations:

[0110] Determine whether the context corresponding to the feature extraction model is available;

[0111] When it is determined that the context corresponding to the feature extraction model is available, step 202 is executed;

[0112] When it is determined that the context corresponding to the feature extraction model is unavailable, a context corresponding to the feature extraction model is created.

[0113] 202. Determine, from the feature extraction model, a feature extraction sub-model corresponding to each type of sub-data to be retrieved in the preprocessed data to be retrieved.

[0114] In an embodiment of the present invention, the feature extraction model includes at least one feature extraction sub-model, and there is a mapping relationship between each feature extraction sub-model and at least one data type, wherein the above mapping relationship can be used to select the feature extraction sub-model adapted to the data type, which is not limited in the embodiment of the present invention.

[0115] 203. For each sub-data to be retrieved, perform a feature extraction operation on the sub-data to be retrieved based on the feature extraction sub-model to obtain a target feature vector corresponding to the sub-data to be retrieved.

[0116] 204. Determine a fusion feature vector corresponding to the data to be retrieved based on the target feature vectors corresponding to all the sub-data to be retrieved.

[0117] 205. Based on the target index structure corresponding to the vector database, determine the vector index corresponding to the fused feature vector.

[0118] 206. Based on the determined retrieval algorithm, a retrieval operation on the vector index is performed in the vector database to obtain a retrieval result corresponding to the data to be retrieved.

[0119] In the embodiment of the present invention, for other detailed descriptions of step 201 and step 203 - step 204, please refer to the detailed description of step 101 and step 103 - step 104 in embodiment 1, which will not be repeated in this embodiment of the present invention.

[0120] It can be seen that the method described in the embodiment of the present invention can, after obtaining the data to be retrieved, perform feature extraction and fusion on the pre-processed data to be retrieved based on the feature extraction model to obtain a fused feature vector, and determine the vector index corresponding to the fused feature vector based on the target index structure corresponding to the vector database, and perform a retrieval operation on the vector index in the vector database based on the determined retrieval algorithm to obtain a retrieval result. It can realize vector retrieval for multimodal data based on the vector database and the feature extraction model, so as to realize multimodal data retrieval based on the semantic content of the data, improve the analysis flexibility and analysis accuracy of unstructured data, improve the analysis flexibility and analysis accuracy of multimodal data, thereby improving data retrieval flexibility and data retrieval accuracy, and then generate accurate and reliable retrieval results, which is conducive to providing users with more efficient retrieval services. In addition, it is also possible to perform feature extraction operations on the sub-data to be retrieved based on the feature extraction sub-model corresponding to each sub-data to be retrieved, obtain the corresponding target feature vector, and then determine the fused feature vector corresponding to the data to be retrieved based on all target feature vectors. This can improve the flexibility and accuracy of the selection of feature extraction methods, thereby improving the feature extraction accuracy and feature extraction flexibility of the data to be retrieved, and further improve the accuracy of the extracted feature vectors, which is conducive to improving the accuracy of the fused feature vector obtained by multimodal feature fusion, so as to further improve the accuracy of data retrieval.

[0121] In an optional embodiment, determining the feature extraction sub-model corresponding to each sub-data to be retrieved in the preprocessed data to be retrieved from the feature extraction model may include the following operations:

[0122] For each sub-data to be retrieved, when the sub-data to be retrieved is image data or video data, determining that the feature extraction sub-model corresponding to the sub-data to be retrieved is an image feature extraction sub-model, and the first feature extraction sub-model adopts a self-attention mechanism;

[0123] For each sub-data to be retrieved, when the sub-data to be retrieved is text data, determining that the feature extraction sub-model corresponding to the sub-data to be retrieved is the second feature extraction sub-model;

[0124] Furthermore, for each sub-data to be retrieved, based on the feature extraction sub-model, a feature extraction operation is performed on the sub-data to be retrieved to obtain a target feature vector corresponding to the sub-data to be retrieved, which may include the following operations:

[0125] For each sub-data to be retrieved, when the sub-data to be retrieved is image data or video data, based on the self-attention mechanism of the first feature extraction sub-model, the semantic features corresponding to the sub-data to be retrieved are extracted as the target feature vector corresponding to the sub-data to be retrieved;

[0126] For each sub-data to be retrieved, when the sub-data to be retrieved is text data, sentence-level features corresponding to the sub-data to be retrieved are extracted based on the second feature extraction sub-model as the target feature vector corresponding to the sub-data to be retrieved.

[0127] Among them, the sentence-level feature is used to represent the feature vector corresponding to a sentence in the data.

[0128] The self-attention mechanism is used to focus more on key areas in image or video data during data processing to better handle long-distance dependencies and capture global information. The self-attention mechanism calculates the attention weights between each element in the data and other elements to achieve global context modeling. The calculation formula of the self-attention mechanism is shown in the following formula:

[0129] Where Q represents query, K represents index and V represents content, d k Represents the dimensions of index and content, and softmax is a normalization operation for similarity.

[0130] Among them, exemplarily, the first feature extraction sub-model can be a convolutional neural network model obtained by improving the model structure based on the Transformer model; the second feature extraction sub-model can be an SBERT (Sentence-BERT) model, which is not limited in the embodiment of the present invention; wherein, the SBERT model is based on the improvement of the BERT (Bidirectional Encoder Representation from Transformers, bidirectional encoding based on Transformer) model, which is suitable for processing sentence-level tasks.

[0131] It can be seen that this optional embodiment can determine that the model corresponding to the sub-data to be retrieved is the first feature extraction sub-model when the sub-data to be retrieved is image data or video data, and extract the semantic features corresponding to the sub-data to be retrieved as the target feature vector; when the sub-data to be retrieved is text data, determine that the model corresponding to the sub-data to be retrieved is the second feature extraction sub-model, and extract the sentence-level features corresponding to the sub-data to be retrieved as the target feature vector, which can further improve the flexibility of selecting the feature extraction model, thereby improving the degree of adaptation between the selected feature model and the data to be retrieved, and further helping to improve the accuracy of the extracted feature vector.

[0132] In another optional embodiment, determining the fused feature vector corresponding to the data to be retrieved based on the target feature vectors corresponding to all the sub-data to be retrieved may include the following operations:

[0133] When the data to be retrieved includes only one type of sub-data to be retrieved, determining the target feature vector corresponding to the sub-data to be retrieved as the fused feature vector corresponding to the data to be retrieved;

[0134] When the data to be retrieved includes at least two sub-data to be retrieved, based on the multimodal fusion strategy corresponding to the multimodal retrieval system, feature fusion processing is performed on the target feature vectors corresponding to all the sub-data to be retrieved to obtain a fused feature vector corresponding to the data to be retrieved.

[0135] Among them, optionally, the multimodal fusion strategy may include a joint representation strategy or a collaborative representation strategy, which is not limited in the embodiments of the present invention; wherein, the joint representation strategy is used to map all target feature vectors to a vector space to fuse multimodal vector features.

[0136] It can be seen that this optional embodiment can directly determine the target feature vector as the fused feature vector corresponding to the data to be retrieved when there is only one type of data to be retrieved. If there are multiple types of data to be retrieved, all target feature vectors are subjected to feature fusion processing based on the multimodal fusion strategy to obtain a fused feature vector, which can improve the flexibility and accuracy of determining the fused feature vector, thereby improving the accuracy of the fused feature vector, which is conducive to completing complex multimodal data retrieval and further improving data retrieval accuracy.

[0137] In an embodiment of the present invention, the flowchart of the data retrieval method may also be shown in FIG6 . When the flowchart of the method is shown in FIG6 , the method may include a data retrieval process and a data storage process. The multimodal data retrieval process may include the following operations:

[0138] Upload the data to be retrieved through the text / image / audio upload interface. After the data upload is completed, input the data to be retrieved into the compression interface to obtain the compressed data to be retrieved. After all the data to be retrieved is compressed, check whether the context (Infer) corresponding to the multimodal retrieval system is available. If not, create a context. If available, perform feature extraction on the data to be retrieved to obtain a fused feature vector. After the feature extraction, release the model context; and perform multimodal search in the vector database based on the fused feature vector to obtain the search results.

[0139] The data storage process may include the following operations:

[0140] The multimodal dataset to be stored is uploaded through the image / audio upload interface. After the data upload is completed, the multimodal dataset is input into the file compression interface to obtain the compressed file compression interface and the compressed file set. After the compression is completed, the compressed file set is stored in the intermediate data storage space (psql, a database shown in Figure 6). When it is determined that the number of compressed files in the intermediate data storage space is greater than the preset number of files, the compressed files are filled into the compressed file queue. If the current compressed file queue is full or timed out, it is determined whether the context (Infer) of the inference model is empty. If it is empty, the context is created. If it is not empty, the compressed file is directly read and the compressed file is input into the model inference interface (i.e., the inference interface). The compressed files are batch processed and the vector dataset to be stored obtained after batch processing is stored in the vector database. If all compressed files are processed and the timeout and compression queue are empty, the context of the inference model is released.

[0141] Example 3

[0142] Please refer to Figure 7, which is a schematic diagram of the structure of a network attached storage device disclosed in an embodiment of the present invention. As shown in Figure 7, the network attached storage device may include:

[0143] A memory 401 storing executable program code;

[0144] a processor 402 coupled to the memory 401;

[0145] The processor 402 calls the executable program code stored in the memory 401 to execute the steps of the data retrieval method described in the first embodiment or the second embodiment of the present invention.

[0146] Example 4

[0147] An embodiment of the present invention discloses a computer storage medium storing computer instructions. When the computer instructions are called, they are used to execute the steps of the data retrieval method described in the first embodiment or the second embodiment of the present invention.

[0148] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the above technical solution, in essence, or the portion that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

Claims

1. A method for data retrieval, characterized in that, The method is applied to a multi-modal retrieval system, which includes a vector database and a pre-trained feature extraction model. The method includes: Obtain the data to be retrieved, where the data to be retrieved includes at least one sub-data to be retrieved; each sub-data to be retrieved includes one of text data, image data, and video data; Based on the feature extraction model, perform a feature extraction and fusion operation on the pre-processed data to be retrieved to obtain a fusion feature vector corresponding to the data to be retrieved; Based on the target index structure corresponding to the vector database, determine the vector index corresponding to the fusion feature vector; Based on the determined retrieval algorithm, perform a retrieval operation on the vector index in the vector database to obtain a retrieval result corresponding to the data to be retrieved.

2. The method for data retrieval according to claim 1, wherein The feature extraction model includes at least one feature extraction sub-model, and there is a mapping relationship between each feature extraction sub-model and at least one of the data types; Among them, the step of performing a feature extraction and fusion operation on the pre-processed data to be retrieved based on the feature extraction model to obtain a fusion feature vector corresponding to the data to be retrieved includes: Determine the feature extraction sub-model corresponding to each sub-data to be retrieved in the pre-processed data to be retrieved from the feature extraction model; For each sub-data to be retrieved, perform a feature extraction operation on the sub-data to be retrieved based on the feature extraction sub-model to obtain a target feature vector corresponding to the sub-data to be retrieved; Determine the fusion feature vector corresponding to the data to be retrieved according to the target feature vectors corresponding to all the sub-data to be retrieved.

3. The method for data retrieval according to claim 2, wherein The step of determining the feature extraction sub-model corresponding to each sub-data to be retrieved in the pre-processed data to be retrieved from the feature extraction model includes: For each sub-data to be retrieved, when the sub-data to be retrieved is the image data or the video data, determine the feature extraction sub-model corresponding to the sub-data to be retrieved as an image feature extraction sub-model, and the first feature extraction sub-model uses a self-attention mechanism; For each sub-data to be retrieved, when the sub-data to be retrieved is the text data, determine the feature extraction sub-model corresponding to the sub-data to be retrieved as a second feature extraction sub-model; And, the step of performing a feature extraction operation on each sub-data to be retrieved based on the feature extraction sub-model to obtain a target feature vector corresponding to the sub-data to be retrieved includes: For each sub-data to be retrieved, when the sub-data to be retrieved is the image data or the video data, extract the semantic feature corresponding to the sub-data to be retrieved based on the self-attention mechanism of the first feature extraction sub-model as the target feature vector corresponding to the sub-data to be retrieved; For each sub-data to be retrieved, when the sub-data to be retrieved is the text data, extract the sentence-level feature corresponding to the sub-data to be retrieved based on the second feature extraction sub-model as the target feature vector corresponding to the sub-data to be retrieved.

4. The method for data retrieval according to claim 2, wherein Determining the fusion feature vector corresponding to the data to be retrieved according to the target feature vectors corresponding to all the sub-data to be retrieved includes: When the data to be retrieved includes only one type of the sub-data to be retrieved, determining the target feature vector corresponding to the sub-data to be retrieved as the fusion feature vector corresponding to the data to be retrieved; When the data to be retrieved includes at least two types of the sub-data to be retrieved, performing feature fusion processing on the target feature vectors corresponding to all the sub-data to be retrieved based on the multi-modal fusion strategy corresponding to the multi-modal retrieval system to obtain the fusion feature vector corresponding to the data to be retrieved.

5. The method for data retrieval according to claim 1, wherein The method further includes: Obtaining the service scenario corresponding to the data to be retrieved and the retrieval requirement corresponding to the data to be retrieved; And, performing a retrieval operation on the vector index in the vector database based on the determined retrieval algorithm to obtain the retrieval result corresponding to the data to be retrieved, including: Performing a retrieval operation on the vector index in the vector database based on the determined retrieval algorithm to obtain a plurality of candidate retrieval results; Analyzing the service relevance of each candidate retrieval result to the service scenario; Screening out at least one target relevance greater than or equal to the preset service relevance from all the service relevances; Determining the candidate retrieval result that meets the retrieval requirement as the retrieval result corresponding to the data to be retrieved according to the candidate retrieval results corresponding to all the target relevances.

6. The method for data retrieval according to claim 1, wherein The method further includes: Obtaining the test vector dimension corresponding to the vector database and the test vector data set corresponding to the test vector dimension; Performing a retrieval operation on the test vector data set corresponding to the test vector dimension in the vector database to obtain the test retrieval result corresponding to the test vector dimension; Analyzing the test retrieval result to obtain the retrieval performance corresponding to the test vector dimension; Adjusting the test vector dimension based on the pre-determined retrieval performance standard and the retrieval performance to obtain the target vector dimension corresponding to the vector database; And, determining the vector index corresponding to the fusion feature vector based on the target index structure corresponding to the vector database, including: Converting the fusion feature vector into the vector index corresponding to the target vector dimension based on the target index structure corresponding to the vector database.

7. The method for data retrieval according to claim 6, characterized in that The method further includes: Obtaining a multi-modal data set to be stored, where the multi-modal data set includes multiple groups of multi-modal data; Performing a compression operation on the multi-modal data set based on the data compression interface corresponding to the multi-modal retrieval system to obtain a set of compressed files corresponding to the multi-modal data set, where the set of compressed files includes multiple compressed files; Extracting a compressed file queue from the set of compressed files; Determining whether the compressed file queue meets the preset data processing condition; When it is determined that the compressed file queue meets the preset data processing condition, performing a batch processing operation on the compressed file queue based on the inference interface corresponding to the multi-modal retrieval system to obtain the vector data set to be stored corresponding to the target vector dimension; Store the vector dataset to be stored in the vector database.

8. The method for data retrieval according to claim 7, wherein The judging whether the compressed file queue meets the preset data processing conditions includes: Obtain the current processing duration of the compressed file corresponding to the multimodal retrieval system; Judge whether the queue length of the compressed file queue is greater than or equal to the preset number of files, whether the context of the inference model corresponding to the multimodal retrieval system is created, and whether the current processing duration of the compressed file exceeds the preset duration; When it is judged that the queue length of the compressed file queue is greater than or equal to the preset number of files, the context of the inference model corresponding to the multimodal retrieval system is created, and the current processing duration of the compressed file does not exceed the preset duration, it is determined that the compressed file queue meets the preset data processing conditions; When it is judged that the queue length of the compressed file queue is less than the preset number of files, or the context of the inference model corresponding to the multimodal retrieval system is not created, or the current processing duration of the compressed file exceeds the preset duration, it is determined that the compressed file queue does not meet the preset data processing conditions.

9. The method for data retrieval according to claim 5, wherein When the service scenario corresponding to the data to be retrieved is a private cloud storage scenario, the method further includes: Determine the sensitive information corresponding to the private cloud storage scenario; Judge whether there is a retrieval result involving the sensitive information among all the candidate retrieval results; When it is judged that there is a retrieval result involving the sensitive information among all the candidate retrieval results, determine the retrieval result involving the sensitive information as the filtered retrieval result; Exclude all the filtered retrieval results from all the candidate retrieval results to obtain the filtered candidate retrieval results; and trigger the operation of analyzing the business relevance between each candidate retrieval result and the service scenario.

10. A network attached storage device, characterized in that, The network attached storage device includes: A memory storing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the data retrieval method according to any one of claims 1-9.

11. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which are used to execute the data retrieval method according to any one of claims 1-9 when the computer instructions are called.

Images