Method, device and product for searching data

Abstract

The invention provides a method, equipment and a product for searching data. The method comprises the following steps: coding search input into a first dense vector based on a first multi-modal search model; based on the first dense vector, determining a distillation data item corresponding to the search input from a distillation data set corresponding to the first multi-modal search model; on the basis of the first multi-modal search model, encoding original data items in an original data subset corresponding to the distillation data items in an original data set corresponding to the distillation data set into second dense vectors; and on the basis of the second dense vector, determining an original data item from the original data subset as a search result corresponding to the search input. According to the method for searching the data, the efficiency and safety of data storage, model reproduction and multi-modal data management can be improved.

Description

Technical Field

[0001] Various embodiments described herein relate to the field of data searching, and more particularly to methods, devices, and computer program products for data searching. Background Art

[0002] The AI ​​or big data industry relies on massive amounts of data and computing power. The management of data, AI / ML (Artificial Intelligence / Machine Learning) models, and underlying IT (Information Technology) systems is complex and costly. Hyper-converged infrastructure (HCI) systems can help alleviate the complexity problem, but traditional technologies such as deduplication (dedup) do little to reduce storage costs.

[0003] As unstructured data increases, people increasingly need multi-modal data management. For example, they want to search for images by text. However, training and maintaining multi-modal deep learning neural networks also requires high costs. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, a device, and a computer program product for data search.

[0005] According to one aspect of the present disclosure, a method for searching data is provided, comprising: encoding a search input into a first dense vector based on a first multimodal search model; determining a distilled data item corresponding to the search input from a distilled data set corresponding to the first multimodal search model based on the first dense vector; encoding an original data item in an original data subset corresponding to the distilled data item in an original data set corresponding to the distilled data set into a second dense vector based on the first multimodal search model; and determining an original data item from the original data subset as a search result corresponding to the search input based on the second dense vector.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processing unit; and a memory coupled to the processing unit and storing instructions, which when executed by the processing unit perform the following actions: based on a first multimodal search model, encode a search input into a first dense vector; based on the first dense vector, determine a distilled data item corresponding to the search input from a distilled data set corresponding to the first multimodal search model; based on the first multimodal search model, encode an original data item in an original data subset corresponding to the distilled data item in an original data set corresponding to the distilled data set into a second dense vector; and based on the second dense vector, determine an original data item from the original data subset as a search result corresponding to the search input.

[0007] According to another aspect of the present disclosure, a computer program product is provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which when executed cause a computer to perform the following operations: based on a first multimodal search model, encode a search input into a first dense vector; based on the first dense vector, determine a distilled data item corresponding to the search input from a distilled data set corresponding to the first multimodal search model; based on the first multimodal search model, encode an original data item in an original data subset corresponding to the distilled data item in an original data set corresponding to the distilled data set into a second dense vector; and based on the second dense vector, determine an original data item from the original data subset as a search result corresponding to the search input.

[0008] The invention summary is provided to introduce relevant concepts in a simplified form, which will be further described in the following detailed description. The invention summary is not intended to identify key features or essential features of the present disclosure, nor is it intended to limit the scope of various embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like elements throughout the exemplary embodiments of the present disclosure.

[0010] Figure 1 An exemplary storage device according to an embodiment of the present disclosure is shown;

[0011] Figure 2 A flowchart of a method for searching data according to an embodiment of the present disclosure is shown;

[0012] Figure 3AA schematic method for constructing a multimodal neural search model according to an embodiment of the present disclosure is shown;

[0013] Figure 3B shows an example of data distillation according to an embodiment of the present disclosure;

[0014] Figure 4A A schematic method for constructing a multimodal neural search model according to an embodiment of the present disclosure is shown;

[0015] Figure 4B A schematic method for constructing a multimodal neural search model according to an embodiment of the present disclosure is shown; and

[0016] Figure 5 A schematic block diagram of a device that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0017] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some specific embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and is not limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure clearer and more complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0018] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless it is explicitly stated that they refer to different objects.

[0019] Take the following embodiments as examples. Although the specification may refer to "an", "one" or "some" embodiments in some places, this does not necessarily mean that each such reference refers to the same embodiment, or that the feature only applies to a single embodiment. Individual features of different embodiments may also be combined to provide other embodiments. In addition, the words "comprise" and "include" should be understood as not limiting the embodiments to consist of only those features that have been mentioned, and such embodiments may also include features / structures that are not specifically mentioned.

[0020] In the field of data search, the term "modal" is short for "data modality". Data modality can be thought of as the type of data. "Multimodality" refers to the ability of a system to use multiple modalities or input channels to achieve the desired goal. For example, humans are able to use vision and hearing at the same time to recognize people or objects. Cross-modality refers to the ability of a system to use information from one modality to improve the performance of another modality. For example, if you see a picture of a dog, you will be able to recognize it as a dog based on the barking when you hear the dog barking. AI systems designed for multiple modalities are called "multimodal" AI systems. However, when referring to AI systems that use information from one modality to improve the performance of another modality, the term "cross-modality" is more accurate.

[0021] Dataset distillation (sometimes also referred to as "data distillation") is the process of creating a small number of synthetic samples from an original dataset that can quickly train a network to (roughly) the same accuracy as when trained on the original dataset. Here, "small amount" means that the amount of data in the dataset obtained after distillation (hereinafter, sometimes referred to as "distilled dataset" or "distilled data") is less than the amount of data in the original dataset, for example, equivalent to 1%-10% (e.g., 5%) of the amount of data in the original dataset, which can be measured, for example, by the number of data items, the size of the storage space occupied, etc. Specifically, for example, in one embodiment, there are 5 million data records (i.e., data items) in the original dataset, and 300,000 data records in the distilled dataset, i.e., the number of data items is 6% of the original dataset. In another embodiment, the storage space occupied by the original dataset is 100MB, and the storage space occupied by the distilled dataset is 5MB, i.e., the storage space occupied is equivalent to 5% of the original dataset.

[0022] It has been shown that for models with known initialization, training the model on a small number of synthetic images from a distribution completely different from the training data can achieve the original accuracy. In other words, the multimodal model trained using the distilled dataset (hereinafter also referred to as the "first multimodal search model") can achieve the same accuracy when processing the task as the multimodal model trained using the original dataset (hereinafter also referred to as the "second multimodal search model") . For example, after training LeNet (a traditional convolutional neural network) on only 10 synthetic images, 94% accuracy can be achieved on MNIST (a handwritten digit recognition task). In view of this, a distilled dataset (each of the distilled data items is an example of the aforementioned "synthetic image") can be used to train a multimodal model, and then the trained multimodal model can be used to complete the recognition task.

[0023] CLIP (Contrastive Language-Image Pre-Training) is a neural network that can efficiently learn visual concepts from natural language supervision. CLIP pretrains an image encoder and a text encoder to predict which images match which texts in a dataset. The method uses a large number of available sources of supervision, such as a large number of texts found on the internet that match images. This data is used to create the following proxy training task for CLIP: given an image, predict which text fragment from a set of 32,768 randomly sampled text fragments in the dataset it actually matches.

[0024] In the present disclosure, the technical problem to be solved is how to improve the efficiency of data storage, model reproduction, and multimodal data management. From a technical point of view, it is necessary to solve technical problems such as how to improve the efficiency of model reproduction and how to build an efficient multimodal neural search engine for multimodal data management. In other words, it is necessary to solve the problem of using a smaller data set (for example, a distilled data set) to obtain a first multimodal search model and the first multimodal search model can reproduce the second multimodal search model obtained using the original data set. Here, "reproduction" means, for example, that, as mentioned above, the second multimodal search model obtained by training with the original data set can achieve 100% accuracy on MNIST; and the first multimodal search model obtained by training with a subset of the original data set or a distilled data set can achieve 94% accuracy on MNIST. In some embodiments, when the accuracy of the first multimodal search model can reach more than 90% of the second multimodal search model, it can be considered that the first multimodal search model can "reproduce" the second multimodal search model. It can be seen that distilled data can reproduce deep learning models with less loss of accuracy.

[0025] In this regard, according to the present disclosure, a hierarchical storage strategy is provided, wherein the distilled data (i.e., the distilled dataset) is stored as an index to the original dataset. The present disclosure also provides an integrated solution for AI and data management, wherein the distilled data can be used to reproduce the AI / ML model, and the distilled information (e.g., the distilled data (i.e., the distilled dataset), or the information obtained based on the distilled data (e.g., a dense vector)) can be stored as an index or data backup in an expensive feature storage (see Figure 1 In the feature storage 130), a multimodal deep learning neural network is constructed on these distilled data (as an index to the original data) to replicate the AI / ML model constructed on the original data to provide a multimodal search feature. Specifically, in some embodiments, a method for searching data is provided, including: based on a first multimodal search model, encoding a search input into a first dense vector; based on the first dense vector, determining a distilled data item corresponding to the search input from a distilled data set corresponding to the first multimodal search model; based on the first multimodal search model, encoding an original data item in an original data subset corresponding to the distilled data item in the original data set corresponding to the distilled data set into a second dense vector; and based on the second dense vector, determining an original data item from the original data subset as a search result corresponding to the search input.

[0026] Through such technical ideas and methods, the present disclosure provides a pipeline processing method that can coordinate AI / ML model training, training data distillation, and multimodal data management, thereby improving the efficiency and security of data storage, model reproduction, and multimodal data management.

[0027] The following references Figures 1 to 5 It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0028] Figure 1 Schematic storage device 100 according to an embodiment of the present disclosure is shown. Figure 1 As shown, the storage device 100 includes a data store 110, a model store 120, and a feature store 130. The data store 110 stores the original data set (for example, Figure 3AThe model storage 120 stores various models, including but not limited to, for example, a data distillation model for performing data distillation and a multimodal search model for searching data (for example, the aforementioned first multimodal search model), and the feature storage 130 stores the distilled data set (for example, Figure 3A The distilled dataset 330 is shown) and its feature information (e.g., the dense vector corresponding to each data item in the distilled dataset).

[0029] Figure 1 The storage device 110 in the embodiment can be regarded as a storage device system obtained by extending the traditional data storage 110 with the model storage 120 and the feature storage 130. In the model storage 120, the feature encoder encodes the distilled data and the user search data (e.g., the search input entered by the user, which can be text or an image) into dense vectors. In the feature storage 130, the dense vectors of the distilled data are indexed as semantic metadata. Then, the data storage product (i.e., the data storage 110) is extended by the model storage 120 and the feature storage 130.

[0030] When a user needs to search for data items that meet a specific search condition from the data storage 110, firstly, a text encoder (e.g., Figure 4A The text encoder 420 shown) or the image encoder (e.g., Figure 4A The image encoder 440 shown encodes the search input into a dense vector, and then uses the nearest neighbors search to find the most similar distilled data item from the distilled data set. The distilled data item is trained from a specific data set in the original data set in the data storage 110, and the specific data set corresponding to the distilled data item is a subset of the original data set (hereinafter, also referred to as the "original data subset"). Then, further based on the distilled data item, the nearest neighbor search is used to find the most similar original data item from the data storage 110, and the original data item is returned to the user as the search result.

[0031] Specifically, if Figure 1 As shown in the curves 141 and 142, the data in the original data set in the data storage 110 is distilled by a data distillation model in the model storage 120 to obtain a distilled data set and stored in the feature storage 130. Figure 1 As shown in the curves 143, 144, 145, and 146, some data items in the original data set in the data storage 110 and some data items in the distilled data set in the feature storage 130 are associated with a model in the model storage 120. Figure 1As shown in the middle curve 147, when the user inputs a search input, the text encoder or image encoder in the model storage 120 is used to encode the search input into a dense vector (hereinafter, also referred to as the "first dense vector"), and the dense vector is compared with each dense vector obtained by encoding the distilled data items in the distilled data set, and the distilled data item most similar to the search input is found from the distilled data set using the nearest neighbor search. That is, the first dense vector is differentiated from each dense vector in the multiple dense vectors (that is, the distance between each dense vector in the multiple dense vectors and the first dense vector), and each differential value is compared to find the smallest difference, and then based on the smallest difference, the dense vector corresponding to the smallest difference in the multiple dense vectors is found, and the corresponding distilled data item in the distilled data set is found based on the dense vector. Further, as Figure 1 As shown in the middle curve 148, based on the distilled data item, each original data item in the original data set corresponding to the distilled data item is encoded into multiple dense vectors (hereinafter, also referred to as "second dense vectors"), and the first dense vector is compared with the second dense vector, that is, the difference between the first dense vector and each second dense vector is calculated. Here, "each original data item in the original data set corresponding to the distilled data item" is also the above-mentioned "specific data set in the original data set". Then, the obtained differential values ​​are compared to find the smallest difference, and then based on the smallest difference, the second dense vector corresponding to the smallest difference is found among the multiple second dense vectors, and based on the second dense vector, the original data item corresponding to the found second dense vector is found among the original data items corresponding to the distilled data item in the original data set, and the original data item is returned to the user as the final search result.

[0032] It should be understood that Figure 1 Only some elements and functional entities are shown, which are all logical units, and their implementation may be different from that shown. It is obvious to those skilled in the art that the storage device 100 generally also includes Figure 1 Other functions and structures than those shown. That is, Figure 1 The storage device 100 is schematically shown as a part of a storage device (eg, a storage product); the embodiment is not limited to the storage device 100 given as an example, but those skilled in the art may apply the scheme to other storage devices having necessary characteristics.

[0033] Figure 2 FIG. 2 is a flowchart of a method 200 for searching data according to an embodiment of the present disclosure. Figure 1 An example method 200 is described.

[0034] like Figure 2As shown, in the exemplary method 200, at 210, based on a first multimodal search model (such as the multimodal search model 350 in FIG. 3), the search input is encoded into a first dense vector. Figure 3A , based on the multimodal search model 350, the search input is encoded into a first dense vector.

[0035] In 220, based on the first dense vector, a distilled data item corresponding to the search input is determined from a distilled data set corresponding to the first multimodal search model (such as the distilled data set 330 in FIG. 3). Figure 3A , based on the first dense vector, determine the distilled data item corresponding to the search input from the distilled data set 330.

[0036] In 230, based on the first multimodal search model, each original data item in the original data subset corresponding to the distilled data item determined in 220 in the original data set corresponding to the distilled data set (such as the original data set 310 in FIG. 3) is encoded into a second dense vector. Here, the original data subset refers to a set of original data items in the original data set corresponding to the distilled data item determined in 220. For example, referring to Figure 3A and Figure 3B , based on the first multimodal search model 350, each original data item in the original data subset corresponding to the distilled data item determined in 220 in the original data set 310 is encoded into a second dense vector respectively.

[0037] In 240, based on the second dense vector, an original data item is determined from the original data subset as a search result corresponding to the search input. For example, the distance (difference) between the first dense vector and each second dense vector is calculated, and the second dense vector with the closest distance (smallest difference) to the first dense vector is found. Then, based on the second dense vector, the original data item corresponding to the second dense vector is determined from the original data subset as a search result corresponding to the search input. For the specific steps of determining the original data item from the original data subset as the search result, reference can be made to Figure 1 Description.

[0038] In order to build a multimodal neural search model, each class of the training dataset is saved as a folder in the data storage, and each file in the folder is a training example of the class. First, a deep learning neural network is trained on the training dataset to obtain a trained deep learning model, and then the training dataset is distilled using dataset distillation to obtain a limited number of distilled images (or sentences) (representing a class) for each data folder. As mentioned above, distilled data can reproduce deep learning models with little loss in accuracy. Distilled data is also used as a representative (index) of data folders to improve the efficiency of data management. Below, refer to Figure 3A, Figure 3B This is explained.

[0039] Figure 3A FIG. 3 shows a schematic method 300A for constructing a multimodal neural search model according to an embodiment of the present disclosure. Figure 3A As shown, the original data set 310 is distilled using the data distillation model 320 to obtain a distilled data set 330. The distilled data set 330 is trained to obtain a first multimodal search model 350. The first multimodal search model 350 is a deep learning model.

[0040] Assume that instead of using the distilled data set 330 to train the search model (i.e., the first multimodal search model 350 shown in FIG. 3 ), the original data set 310 is used to train the search model (i.e., the second multimodal search model 340 shown in FIG. 3 ). In this case, as mentioned above, the first multimodal search model 350 can achieve comparable performance to the second multimodal search model 340. For example, the accuracy of using the first multimodal search model 350 to process a task can reach more than 90% of that when the second multimodal search model 340 is used to process the same task. Therefore, it can be considered that the first multimodal search model 350 can "reproduce" the second multimodal search model 340. That is, distilled data can reproduce the deep learning model with less loss of accuracy. Therefore, the first multimodal search model 350 (rather than the second multimodal search model 340), which is lighter than the second multimodal search model 340, can be used to process the search task. In view of this, Figure 3A The arrowed line from the original data set 310 to the second multimodal search model 340 is shown as a dotted line, and the arrowed line from the distilled data set 330 to the first multimodal search model 350 is shown as a solid line, to indicate that in actual applications, the lighter first multimodal search model 350 (rather than the second multimodal search model 340) can be directly used to handle search tasks.

[0041] Figure 3B FIG. 3 shows a data distillation example 300B according to an embodiment of the present disclosure. In the data distillation example 300B, a distilled data set 370 is obtained by distilling the original data set 360. The distilled data set 370 is Figure 3A The distilled dataset 330 is also an example of a distilled dataset. The original dataset 360 is Figure 3A The original data set 310 is also an example of an original data set. Figure 3B It can also be seen that the amount of data in the distilled dataset 370 is much smaller than that in the original dataset 360.

[0042] In some embodiments, for example, in the original data set 360, as described above, there are 10 classes, and the class names are "airplane", "car", "bird", "cat", "deer", "dog", "frog", "horse", "ship", and "truck". Each class is stored in the data store (e.g., Figure 1 The data storage 110 is stored in a folder, and each file in the folder is a training example of the class. Figure 3B In the illustrated embodiment, there are 5,000 images (i.e., 5,000 image data items) in each class, and there are 50,000 images (i.e., 50,000 image data items) in total. In other words, 5,000 image files are stored in the folder corresponding to each class, and there are 10 folders corresponding to a total of 10 classes, that is, a total of 50,000 image files are stored in these 10 folders.

[0043] Train a deep learning neural network on the training dataset to obtain a trained deep learning model, and then use data distillation to distill the training dataset to obtain a limited number of distilled images (or sentences) (representing a class) for each data folder. Figure 3A In the illustrated embodiment, data distillation is performed on the original data set 360 (see Figure 3A 320 in the figure) to obtain a distilled dataset 370. In the obtained distilled dataset 370, there are 50 images in each class, and there are 500 images in 10 classes. It can be seen that the data volume of the distilled dataset 370 is 1% of the original dataset 360.

[0044] After obtaining the distilled data set 370, all distilled data (i.e., all distilled data items in the distilled data set 370) are encoded, and the encoded dense vectors are stored in a feature storage (e.g. Figure 1 The feature storage 130 is shown as an index into the original data set 360 .

[0045] In some embodiments, when using Figure 1 When the storage device 100 performs data search, in the first stage, a multimodal neural search model (e.g., Figure 3A The first multimodal search model 350 is shown).

[0046] Figure 4AA schematic method 400A for constructing a multimodal neural search model according to an embodiment of the present disclosure is shown. For example, for an image in an original data set, distillation is performed to obtain distilled image data (hereinafter, also referred to as "distilled image data"), and the distilled image data includes text information and image information. The text information includes, for example, a machine learning task description, a data set description, a class name, and a folder path, and the image information includes, for example, a distilled image (hereinafter, also referred to as a "distilled image"). When constructing a multimodal neural search model, the text information included in the distilled image data (i.e., the machine learning task description, the data set description, the class name, and the folder path, etc.) is used as the text input 410 of the CLIP model and the distilled image is used as the image input 430 of the CLIP model to train the multimodal neural search model. Specifically, as Figure 4A As shown, text input 410 is input into text encoder 420 for encoding to obtain text information (such as Figure 4A As shown in T1, T2, T3, ..., T N ), the image input 430 is input into the image encoder 440 for encoding to obtain image information (such as Figure 4A As shown in I1, I2, I3, ..., I N Then, in the distillation of the image and its overall description pair (pair, such as Figure 4A As shown in I1T1, I1T2, I1T3, ..., I1T N ,I2T1,I2T2,I2T3,…,I2T N ,……I N T N ) to train the CLIP model.

[0047] Figure 4B Another exemplary method 400B for constructing a multimodal neural search model according to an embodiment of the present disclosure is shown. Figure 4B The method shown is Figure 4A The difference of the method shown is that for the text files in the original data set, since they do not contain image information, distillation is performed to obtain a distilled text file, which only includes text information but not image information. The text information includes, for example, machine learning task description, data set description, class name and folder path. Therefore, when constructing a multimodal neural search model, the distilled text file itself and its text information (for example, machine learning task description, data set description, class name and folder path, etc.) are used as the text input 450 of the CLIP model to train the multimodal neural search model, and only the text encoder of the CLIP model is trained as a language model. Specifically, in Figure 4B In the example, only the text input 450 is input to the text encoder, wherein the text input 450 is Figure 4AThe difference between text input 410 and text input 450 is that, in addition to text information (e.g., machine learning task description, dataset description, class name, folder path, etc.), text input 450 also includes the distilled text file itself.

[0048] In use Figure 1 When the storage device 100 performs data search, in the second stage, the multimodal neural search model (e.g., Figure 3A The first multimodal search model 350 is shown in FIG. 3 , and the user performs a multimodal search. In this stage, first, the user enters a search input, which may be, for example, a search keyword (text) or a picture (image). The search input is encoded into a dense vector (first dense vector) using the multimodal neural search model constructed in the first stage, and the nearest neighbor search is used to extract the first dense vector from the distilled data (e.g., Figure 3A Each distillation data item in the distillation data set 330 is shown, Figure 3B The distilled data items in the distilled data set 370 shown in the figure are encoded) to find the dense vector that is most similar to the first dense vector (that is, the dense vector with the smallest distance to the first dense vector), and then find the distilled data item corresponding to the most similar dense vector.

[0049] Then, the multimodal neural search model is used to search the original dataset (e.g. Figure 3A The original data set 310 is shown. Figure 3B All the original data items represented by the most similar distilled data item in the original data set 360 shown in the figure are encoded to obtain multiple dense vectors (second dense vectors), and the dense vector most similar to the first dense vector is found from these second dense vectors using the nearest neighbor search, and then based on the most similar dense vector, the original data item corresponding to the most similar dense vector is found from the original data items represented by the most similar distilled data item (the original data item is the original data item most similar to the search input entered by the user) as the search result corresponding to the search input entered by the user. Here, the "original data item represented by the most similar distilled data item in the original data set" refers to each original data item in the original data set corresponding to the distilled data item, which is also referred to as the "specific data set in the original data set" in this article, which refers to, for example, the original data item in the original data set that is similar to the one in the search input entered by the user. Figure 2 The original data items in the original data subset corresponding to the distilled data items determined in 220 are shown. In the data distillation process, the original data items in the original data set are distilled to obtain the distilled data items.

[0050] Figure 5Schematic block diagram of a device 500 that can be used to implement an embodiment of the present disclosure is shown. The device 500 may be a device, apparatus, or system described in an embodiment of the present disclosure. For example, the device 500 may be any hardware equipped with the service 120 of the present disclosure, such as a server, a device (such as a terminal device), etc. Figure 5 As shown, the device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 502 or computer program instructions loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0051] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0052] The various methods or processes described above may be performed by the processing unit 501. For example, in some embodiments, the methods may be implemented as computer software programs, which are tangibly contained in a machine-readable medium, such as the storage unit 508. For example, in some embodiments, the service 120 (or, more specifically, the methods implemented therein) may be implemented as computer software programs, which are tangibly contained in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the CPU 501, one or more steps or actions in the methods or processes described above may be performed.

[0053] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0054] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0055] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0056] The computer program instructions for performing the disclosed operation can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages, and the programming languages ​​include object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partly on the user's computer and partly on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or, can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In certain embodiments, by utilizing the state information of computer-readable program instructions to customize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0057] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0058] Computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0059] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the equipment, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each block in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the blocks can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous blocks can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flow chart, and the combination of the blocks in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0060] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for searching data, comprising: Based on the first multimodal search model, encoding the search input into a first dense vector; Based on the first dense vector, determining a distilled data item corresponding to the search input from a distilled data set corresponding to the first multimodal search model; Based on the first multimodal search model, encoding original data items in an original data subset corresponding to the distilled data items in an original data set corresponding to the distilled data set into a second dense vector; and Based on the second dense vector, an original data item is determined from the original data subset as a search result corresponding to the search input.

2. The method of claim 1 , wherein before encoding the search input into the first dense vector, further comprising: Determine the distilled data set by distilling the original data set; Based on the distilled dataset, the first multimodal search model is constructed.

3. The method of claim 1, wherein the distilled dataset reproduces a second multimodal search model constructed based on the original dataset.

4. The method according to claim 2, wherein each distilled data item included in the distilled dataset is distilled image data or a distilled text file. 5 . The method according to claim 1 , wherein the class of the original data set is saved as a folder in a data storage, and the files in the folder are training examples of the class.

6. The method according to claim 4, wherein constructing the first multimodal search model comprises one of the following: In a case where the distilled data item is the distilled image data, using the task description, dataset description, class name, and folder path corresponding to the distilled image data as text input, and using the distilled image corresponding to the distilled image data as image input to train the first multimodal search model; or In the case where the distilled data item is the distilled text file, the first multimodal search model is trained using the distilled text file, the task description, dataset description, class name and folder path corresponding to the distilled text file as text input.

7. The method of claim 1, wherein the distilled data items in the distilled data set are used as indexes to the original data items in the original data set.

8. The method according to claim 2, wherein constructing the first multimodal search model further comprises: Encoding the distilled data items in the distilled data set to obtain multiple dense vectors; as well as The plurality of dense vectors are stored as indices to original data items in the original data set.

9. The method of claim 8, wherein determining the distillation data item comprises: Determine, from the plurality of dense vectors, a target dense vector that is closest to the first dense vector; as well as A target distilled data item corresponding to the target dense vector is determined from the distilled data set as the distilled data item.

10. The method of claim 9, wherein the distance is a Euclidean distance.

11. The method of claim 9, wherein encoding the original data item into the second dense vector comprises: Determine the original data subset corresponding to the target distilled data item in the original data set; as well as The original data items in the original data subset are encoded into the second dense vectors through the first multimodal search model.

12. The method of claim 11, wherein determining the original data item comprises: Determine, from the second dense vector, a second dense vector that is closest to the first dense vector; as well as An original data item corresponding to the second dense vector with the shortest distance is determined from the original data subset.

13. An electronic device comprising: Processing unit; as well as A memory coupled to the processing unit and storing instructions that, when executed by the processing unit, perform the following actions: Based on the first multimodal search model, encoding the search input into a first dense vector; Based on the first dense vector, determining a distilled data item corresponding to the search input from a distilled data set corresponding to the first multimodal search model; Based on the first multimodal search model, encoding original data items in an original data subset corresponding to the distilled data items in an original data set corresponding to the distilled data set into a second dense vector; and Based on the second dense vector, an original data item is determined from the original data subset as a search result corresponding to the search input.

14. The electronic device of claim 13, wherein before encoding the search input into the first dense vector, the instructions, when executed by the processing unit, further perform the following actions: Determine the distilled data set by distilling the original data set; Based on the distilled dataset, the first multimodal search model is constructed.

15. The electronic device of claim 14, wherein the distilled dataset reproduces a second multimodal search model constructed based on the original dataset. 16 . The electronic device according to claim 14 , wherein each distilled data item included in the distilled data set is distilled image data or a distilled text file.

17. The electronic device according to claim 14, wherein constructing the first multimodal search model comprises one of the following: In a case where the distilled data item is the distilled image data, the first multimodal search model is trained using a task description, a dataset description, a class name, and a folder path corresponding to the distilled image data as text input and a distilled image corresponding to the distilled image data as image input; or In the case where the distilled data item is the distilled text file, the first multimodal search model is trained using the distilled text file, the task description, dataset description, class name and folder path corresponding to the distilled text file as text input.

18. The electronic device of claim 13, wherein encoding the original data item into the second dense vector comprises: Determining the original data subset corresponding to the distilled data item in the original data set; as well as The original data items in the original data subset are encoded into the second dense vectors through the first multimodal search model.

19. The electronic device of claim 18, wherein determining the original data item comprises: Determine, from the second dense vector, a second dense vector that is closest to the first dense vector; as well as An original data item corresponding to the second dense vector with the shortest distance is determined from the original data subset.

20. A computer program product tangibly stored on a non-transitory computer readable medium and comprising computer executable instructions that, when executed, cause a computer to: Based on the first multimodal search model, encoding the search input into a first dense vector; Based on the first dense vector, determining a distilled data item corresponding to the search input from a distilled data set corresponding to the first multimodal search model; Based on the first multimodal search model, encoding original data items in an original data subset corresponding to the distilled data items in an original data set corresponding to the distilled data set into a second dense vector; and Based on the second dense vector, an original data item is determined from the original data subset as a search result corresponding to the search input.

Images