Unified framework for multi-modal similarity search

Abstract

Embodiments of the present disclosure relate to a unified framework for multi-modal similarity search. Disclosed herein is a technique for enhanced similarity search. In implementations, a search environment includes one or more computing hardware, software, and / or firmware components to support enhanced similarity search. The one or more components identify a modality of a similarity search relative to the query object. The component generates an embedding for the query object based on the modality and based on connections between the query object and neighboring nodes in the graph. The embedding of the query object provides a basis for the search of similar objects.

Description

[0001] This application is a divisional application of the application with application date of February 5, 2021, application number 202110163131.1, and invention name “Unified Framework for Multimodal Similarity Search”. Technical Field

[0002] Various aspects of the present disclosure relate to the fields of computing hardware, software, and services, and in particular to similarity search techniques. Background Art

[0003] Similarity search typically works by comparing the similarity between pairs of objects. For example, a dataset can be searched for the top k objects most similar to a query object, based on a comparison of each object in the collection with the query object. Thus, a unique pair of objects consisting of a query object and each object in the collection can be evaluated based on the similarity of the target object in a given pair to the query object. The similarity of one object to another can be evaluated based on visual similarity, conceptual similarity, or any other modality, and the search pipeline can be optimized for any desired modality.

[0004] The search pipeline generally consists of a feature extraction phase (or embedding) followed by an evaluation phase. During the first phase, features of the query object are extracted and fed as input to the evaluation phase. In the evaluation phase, the features of the query object are compared with the features of the target object to find the most similar objects. Feature extraction can be weighted toward one dimension or another depending on the selected optimization, just as feature evaluation can be weighted. For example, the search for objects that are most visually similar to the query object can be optimized for the visual features of the object, while the search for conceptually similar objects can be optimized for conceptual (or textual) features.

[0005] A drawback of existing optimization techniques is that similarity searches focused on one modality or dimension often miss objects that may be similar to other modalities. Another drawback (at least relative to search solutions that employ machine learning models) is that each type of optimization requires a separate model. That is, two models optimized for different modalities will be trained on different datasets in order to produce the desired optimization. Therefore, current optimizations lack fine-grained control over similarity and lack scalability with respect to adding new modalities. This constraint makes search optimization a tedious task, especially relative to those that employ machine learning models.

[0006] Machine learning models can be employed in the context of similarity searches in either or both the feature extraction and evaluation phases. During feature extraction, a query object can be input to a model that produces an embedding for that object. The embedding can then be passed to any type of search tool—whether utilizing a machine learning model or a traditional rule-based approach. The search tool compares the embedding of the query object with the embeddings of objects in the target dataset to identify and return objects that are considered to be the top k most similar objects in the set.

[0007] Some solutions use models trained on a graph of existing objects (e.g., GraphSAGE) to produce an embedding for a query object. A graph consisting of nodes and edges representing objects and their interconnections takes a query object as input and produces an embedding for the object as output. The embedding can then be passed to a search tool to find its k nearest neighbors in a set of existing objects. By selecting training data that emphasizes one modality or the other, the model can be optimized for one modality or the other. Unfortunately, as discussed above, this technique requires a different model for each optimization and is neither easily interpretable nor scalable.

[0008] In a simple example, an existing image collection can be queried to find images similar to the query image. To search existing images, a machine learning model is trained on node embeddings using a graph. The graph consists of nodes representing images and edges representing their connections. For example, an image can be connected to other images in the collection that are similar to the given image. Images of cats will be connected to other images of cats; images of dogs will be connected to other images of dogs; images of cats and dogs will be connected to images of cats and dogs, and so on.

[0009] The nodes contain embeddings, or feature vectors, that encode information about each image. A machine learning model is trained on the graph to generate embeddings for new input images: the trained model takes a query image as input and produces an embedding of the image as output. The embedding of a given query image can then be used in similarity searches across the dataset to find images that are similar to the query image. Unfortunately, once a model has been trained, its modality cannot be changed, and the model will always generate embeddings based on the single modality it was trained on. Summary of the Invention

[0010] This paper discloses a unified framework for similarity search that allows multiple modalities to be jointly modeled so that the dimensions to be emphasized in the search can be selected on a per-search basis. This framework can be extended to any number of dimensions, and control over modality selection can be explicit, implicit, or otherwise. The modality selected for a given search affects one or both of the following: which edges of the query object are input to the model that produces the embedding of that object; and the weighting applied to the embedding. The resulting embedding intentionally biases the similarity search towards the desired modality.

[0011] In various implementations, the search environment includes one or more computing hardware, software, and / or firmware components to support enhanced similarity search. In operation, one or more components identify a modality for similarity search relative to a query object. The component generates an embedding of the query object based on the modality and the connections between the query object and neighboring nodes in a graph. The embedding of the query object provides a basis for searching for similar objects.

[0012] In some implementations, the graph is used to train a machine learning model that produces an embedding for a query object. The embedding produced by the model may be biased by the input supplied to the model. In some cases, one or more components selectively identify which edges to include with the query object in the model input to bias the resulting embedding toward one of multiple possible modalities. The embedding output by the model can be provided to a search component to find similar objects.

[0013] In the same or other implementations, the output obtained from the machine learning model can be considered as an initial embedding for the query object. This initial embedding can be combined with the label embedding to form a combined embedding that is input to the similarity search. The initial embedding and the label embedding can be combined based on a weight specified by the modality, so that the combined embedding is biased towards the selected modality, thereby affecting the final result of the search.

[0014] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. It should be understood that this Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Many aspects of the present disclosure may be better understood with reference to the following drawings. The components in the drawings are not necessarily drawn to scale, emphasis instead being placed on clearly illustrating the principles of the present disclosure. Furthermore, in the drawings, like reference numerals designate corresponding parts throughout the various views. Although several embodiments have been described in conjunction with these drawings, the present disclosure is not limited to the embodiments disclosed herein. Rather, it is intended to cover all alternatives, modifications, and equivalents.

[0016] Figure 1 The diagram illustrates the search environment in implementation.

[0017] Figure 2 The diagram illustrates the search process in the implementation.

[0018] Figure 3 The diagram illustrates the embedding process in implementation.

[0019] Figures 4A to 4B The search environment in the implementation and two operational scenarios are illustrated respectively.

[0020] Figure 5 The diagram illustrates the embedding process in implementation.

[0021] Figures 6A to 6B The search environment in the implementation and two operational scenarios are illustrated respectively.

[0022] 7A to 7D An exemplary similarity search conducted via two different modalities in an implementation is illustrated.

[0023] Figure 8 A computing system suitable for implementing the various operating environments, modules, architectures, processes, scenarios, and sequences discussed below with respect to the figures is illustrated. DETAILED DESCRIPTION

[0024] Optimizing similarity search for the desired modality is a well-known challenge in representation learning, particularly relative to visual search. Optimizing solely for visual similarity produces results that lack diversity relative to conceptual similarity, just as optimizing for conceptual similarity produces results that lack visual similarity. Furthermore, optimizing for similarity search for one modality relative to another can limit or completely suppress the explanatory value and power of the results. Adding new dimensions to the similarity search framework is also difficult.

[0025] The unified framework disclosed herein proposes representation learning on multimodal graphs that include objects and their metadata (labels). The graph can be jointly modeled in at least two modalities (e.g., visual and conceptual) so that the resulting model can produce embeddings based on multiple modalities. In addition, reasoning edge connections provide explanatory value and provide control over which dimension to select for similarity search (e.g., visual or conceptual). This control can be implicit based on user intent, such as a search query, or explicit using a user interface to accept a modality selection. Selectively identifying connections to the graph provides the ability to control the search mode.

[0026] Dimensions need not be limited to visual or conceptual, but can be extended to any dimension supported by available metadata in the graph structure or machine learning inference. In addition, the unified framework is applicable to a variety of data, including images, documents, videos, biological structures, chemical structures, etc.

[0027] In various implementations disclosed herein, a graph is proposed that includes a combination of nodes with different modalities and edges between them to establish associations. Examples include objects, object labels, and connections determined based on similarity. The graph connects each object with its corresponding label and the k nearest neighbors of a given object. Labels can be part of the object (such as text embedded in an image), or they can include short sequences of words that describe the content of the object.

[0028] An object's k nearest neighbors are determined based on their similarity scores along the first dimension to the query object for all objects in the corpus (ranked closest, hence "nearest"). An example dimension is the "visual" similarity of images determined by a separate deep neural network trained and optimized to learn to represent images in a high-dimensional space where similarity can be captured by a distance metric (closest in distance are more similar). This visual similarity metric for nearby edges in a graph is just one example of how the framework can be generalized to any similarity metric between nodes of the same type or between different node types.

[0029] Once the multimodal graph has been constructed, a model is trained on the graph to convert each node into an embedding (feature vector) that contains one dimension of information (e.g., visual) and at least a second dimension of information (e.g., conceptual) extracted through the graph connections. In some implementations, object nodes in the graph are initialized with ResNet-18 features of size 512. Label nodes are randomly initialized with label embeddings of size 512. The training seeks to reduce the distance between directly connected node embeddings and increase the distance between node embeddings that are more than one hop apart. The trained model can then be deployed with the support of joint representation learning. That is, the model can be used for similarity search in any of the multiple modalities for which it was trained.

[0030] This model can be used for at least two different purposes. In one purpose, the model can be used to predict the label of a query object. Using images as an example, a given query image is connected to the top k images in the graph based on the minimum distance. The query image and its connections are then fed into the model to generate an embedding for the query node. The top k labels can then be found based on the minimum distance between the query node embedding and the label embeddings of the labels in the graph.

[0031] This model can also be used for object search and retrieval. When no label is available for the query object, the first method uses object-object connections. In this case, the query object is connected to the top k closest objects in the graph to generate a new query embedding. Similar objects can then be found based on the minimum distance comparison between the query object embedding and the object embeddings in the graph.

[0032] In a variation, object-label nodes can be utilized in addition to object-object nodes. The query object is again connected to the top k objects in the graph, but is also connected to its corresponding label node in the graph. Based on the query object, its object-object connections, and its object-label connections, a new query embedding is generated. The resulting query embedding can be used to find the top k similar objects in the graph. In an alternative approach, object-label nodes can be used exclusively for generating new query embeddings.

[0033] Figure 1 The search environment 100 in the implementation of a unified framework for multimodal similarity search is illustrated. The search environment 100 includes a model training component 101, an embedding component 111, and an evaluation component 121. The model training component 101, the embedding component 111, and the evaluation component 121 work together to provide enhanced similarity search capabilities. The search environment 100 can be implemented in hardware, software, and / or firmware and in the context of one or more computing systems. Figure 8 The computing system 801 in FIG. 8 is representative of the one or more computing systems.

[0034] Broadly speaking, search environment 100 takes as input a query object, represented by query object 113, and identifies multiple objects represented by search results 125 that are similar to query object 113. Search environment 100 accomplishes this task by generating an embedding of the query object, represented by query embedding 115, using a joint model, represented by model 110, trained on graph 107.

[0035] More specifically, model training component 101 jointly trains model 110 on graph 107, which includes nodes and edges corresponding to existing objects, labels, and their connected datasets 105. Examples of existing objects include, but are not limited to, images, documents, videos, audio, molecules, and proteins. Labels include textual descriptions or tags of at least some objects. For example, in an image collection, labels may identify the context or content of a single image, while in a document collection, labels may describe the types of documents in the collection.

[0036] The nodes in graph 107 correspond to objects and labels, and the edges in graph 107 correspond to their connections. Object pairs that are sufficiently similar to each other are connected by edges, and labeled objects can be connected to label nodes by edges. Although some objects can be connected to one or more labels, a given object does not need to be connected to any label at all, although each object is connected to at least one other object by an edge. Whether two objects are similar enough to be connected is a design problem based on empirical evidence and can be considered to be beyond the scope of this disclosure. In some cases, graph 107 already exists and its construction need not be worried about. In addition to nodes and edges, graph 107 also includes embeddings (feature vectors) that encode information about each object and label. Existing embeddings can also be determined a priori and, together with nodes and edges, include training data for training model 110.

[0037] The model training component 101 jointly trains a model 110 on the graph 107 and its embedding associated with the dataset 105. The model 110 is shown in an untrained state (indicated by no fill color) in the model training component 101 and in its trained state (indicated by black fill) in the embedding component 111. The model 110 can include one or more artificial neural networks composed of layers of neurons. These layers are connected to each other in various ways, and the neurons fire in response to various inputs. The input at the first layer is the data being analyzed (e.g., the graph 107). The output at the final layer represents what the neural network has learned or recognized (e.g., an embedding). In between are intermediate or "hidden" layers that perform additional processing.

[0038] Each neuron has a rule that tells it to fire (or not fire) based on its inputs. The inputs are weighted, and the weights can change based on a feedback process. In the case of graph data, the inputs to the first layer(s) of the artificial neural network are vectors filled with numerical or symbolic descriptions of the nodes and edges in the graph that represent objects, labels, and connections in the existing dataset. The inputs to the hidden layers are the outputs of the neurons in the first layer. Within the hidden layers, the outputs from one layer are the inputs to the subsequent layers. The last layer of the artificial neural network takes as input the outputs of the last hidden layer and outputs the value that identifies what has been learned as the signal passes through the layers of neurons.

[0039] During the training phase of a neural network, the weights associated with the neurons are changed based on the difference between the results produced by the neural network and the correct embedding of the neural network input. Over time, and after many feedback cycles, the weights are configured so that the neural network can accurately produce embeddings for new objects. The final configuration of weights and neurons is considered a model of the subject under scrutiny (such as an object and its embedding).

[0040] Embedding component 111 utilizes model 110 in its trained state to generate an embedding for a query object, such as query object 113. Embedding component 111 includes a modality selector 112, which includes a subcomponent capable of determining a search modality, which will take into account how embedding component 111 generates embeddings. Modality selector 112 can determine the modality explicitly (e.g., from user input or other such instructions), implicitly, or otherwise. The resulting query embedding 115 includes numerical and / or symbolic information having characteristics of query object 113 encoded therein.

[0041] Embedding component 111 passes query embedding 115 to evaluation component 121 for use as input to search function 123. Search function 123 searches the dataset for objects similar to query object 113 by comparing query embedding 115 with the embeddings of objects in the dataset. Search function 123 can employ machine learning methods for finding similar objects, rule-based methods, a combination of both, or any other type of search technology capable of utilizing query embedding 115. The target dataset can be dataset 105 and its embedding, although it is possible to search datasets other than dataset 105. The output of evaluation component 121 includes one or more objects represented by similar objects, which, as mentioned, can be obtained from dataset 105.

[0042] To implement the enhanced similarity search discussed above, the one or more computing systems providing the search environment 100 execute a search process 200 within the context of the components of the search environment 100 . Figure 2 The illustrated search process 200 may be implemented as program instructions in the context of any of the following: hardware, software applications, modules, or other such programming elements including the model training component 101, the embedding component 111, and the evaluation component 121. Figure 2 In the steps of , the program instructions direct (multiple) of its host computing systems to operate as described for search process 200.

[0043] In operation, the search process 200 obtains a graph of existing objects in a dataset (step 201). The graph includes nodes and edges corresponding to the objects and labels in the dataset, and their interconnections. The search process 200 then proceeds to train a machine learning model on the graph and the embeddings of the objects in the graph (step 203). The embeddings can be included in the graph, although they can also be provided separately.

[0044] Next, the search process 200 identifies modalities for similarity searches of objects in the graph relative to a query object (step 205). Examples of query objects include images, documents, proteins, and other such objects for which embeddings can be created. Thus, examples of datasets to search include image repositories, document repositories, protein repositories, and the like. The modalities can be selected from a range of possible modalities that differ from one another with respect to how they affect the embedding generated for a given query object. More specifically, at least one of the possible modalities will encode the connection of the query object to its labels in its resulting embedding to a greater extent than other modalities, and some of the modalities may completely ignore the object-label connection.

[0045] For example, with respect to image data, the modality can be selected from at least a visual modality and a conceptual modality. The visual modality will bias the embedding toward visual characteristics or attributes of the query object, while the conceptual modality will bias the embedding toward conceptual characteristics or attributes of the query object. Examples of visual characteristics include, but are not limited to, spectral mean, spectral minimum, spectral maximum, etc. Examples of conceptual attributes are related to the connection(s) between the query image and one or more labels in the dataset, as labels describe the content of the image to which they are connected.

[0046] Similar concepts apply to documents, proteins, and other searchable objects. For example, documents can also be searched via a visual modality that emphasizes the visual attributes of the query document compared to when using a concept-oriented modality. When using a concept modality, the embedding generated for the query document will encode its document-label connection more robustly than achieved by other modalities (especially those that ignore document-label connections entirely).

[0047] It will be appreciated that the visual modality is only one example of a variety of possible modalities. Other examples include content-based modalities, which take into account the properties of the content (words, phrases, etc.) contained in a document or other such object, or the properties of the chemical composition of the molecules or materials of the object being searched. In any case, the signature difference between any two modalities is that one modality is able to encode the object-label connection of the query object to a greater extent than the other modality.

[0048] The identified modality need not be an either-or selection. Rather, the identified modality may reside anywhere within the continuum of modalities. The modality may be determined by user input comprising selecting the modality from a set of possible modalities in a user interface. Alternatively, the user input may be a position on a sliding scale between two modalities (e.g., a "slider" that can be positioned anywhere between two different modalities).

[0049] With the modality identified, the search process 200 continues by generating an embedding for the query object based on the identified modality and connections of the query object to the graph (step 207). The connections to the graph may include one or both of object-object connections and object-label connections. The connections may optionally include connections between the query object and other types of nodes that may be included in the graph in addition to object nodes and label nodes.

[0050] The resulting embedding can be biased toward the selected modality in a variety of ways. In some implementations, the bias is accomplished by selecting certain connections from the query object to the graph relative to other connections. For example, a visual search will select object-object connections and will ignore or reduce object-label connections. In contrast, a conceptual search will include more object-label connections than a visual search. In the sliding scale example, the resulting modality will include object-label connections to a greater extent the closer the user input moves the slider toward a pure conceptual search, and to a lesser extent (if any) the further the slider moves away from a conceptual search.

[0051] In other implementations, embeddings can be biased in different ways, generating an initial embedding for the query object, which is then combined with the embeddings generated for the associated tags. The combination of the initial embedding and the tag embedding results in an embedding for the query object, and the combination step can be weighted based on the determined modality. For example, the weighting can emphasize or favor tag embeddings for conceptual modalities while de-emphasizing tag embeddings for modalities that are less conceptual (e.g., visual modalities). A sliding scale concept can also be used in such an embodiment. In a sliding scale implementation, the closer the user input causes the slider to slide toward the end representing the conceptual modality, the more the weighting will weight the tag embedding. The farther the slider is from the conceptual modality, the less weight is given to the tag embedding on a relative basis.

[0052] The search process 200 obtains the resulting embedding of the query object and supplies it as input to a similarity search function (step 209). The similarity search function can be, for example, a k-nearest neighbor search that compares the query embedding with the embeddings of objects in the target dataset. The similarity search function can utilize machine learning methods, rule-based methods, or any combination or variation thereof to search for objects similar to the query object based on their embeddings (step 211). The search results can be returned to any application, service, or component that originally requested the search.

[0053] Applying the search process 200 to elements of the search environment 100, the components of the search environment 100 execute as follows. It will be appreciated that although the steps of the search process 200 may be attributed to specific components in the search environment 100, they may be performed by other components. Figure 1 The search environment 100 may be executed by one or more of the components illustrated, as well as other components not shown or those external to the search environment 100 .

[0054] In operation, the model training component 101 obtains a graph 107 of existing objects in the dataset 105. The graph 107 includes nodes and edges corresponding to the objects and labels in the dataset, and their interconnections. The model training component 101 then proceeds to train a model 110 on the graph and the embeddings of the objects in the graph, and provides the model 110 in its trained state to the embedding component 111.

[0055] A similarity search is then initiated relative to the query object 113. The modality selector 112 determines the modality of the search, which informs the embedding component 111 how to generate an embedding for the query object 113. The modality may be determined explicitly, implicitly, or otherwise through a set of possible modalities, ranges, or others.

[0056] Embedding component 111 continues to generate an embedding for the query object through one path or a possible combination of paths. In some scenarios, embedding component 111 selectively identifies connections between the query object and other objects in the graph based on the determined modality. The query object and its connections are then input to model 110 to generate an output comprising query embedding 115. In a simplified example, embedding component 111 can include object-object connections and object-label connections according to some modalities, while excluding or restricting object-label connections in other modalities.

[0057] In other scenarios, embedding component 111 can generate an initial embedding for a query object based on any modality. As in the scenario just discussed above, embedding component 111 provides the query object and at least some of its connections to model 110 to generate an initial embedding. This initial embedding is then combined with the tag embeddings generated for the tags associated with the query object. In this scenario, query embedding 115 is the result of combining the initial embedding and the tag embeddings. The tag embeddings can be known a priori or generated by model 110.

[0058] In either case, the query embedding 115 is submitted to an evaluation component 121. The evaluation component 121 provides the query embedding 115 as input to a search function 123, which can compare the query embedding with embeddings associated with the dataset 105, whether using a machine learning approach, a rule-based approach, or any other type of search. Given the modality of the search, the results output by the search function 123 are the top k objects that are considered most similar to the query object 113. The results can be, for example, images similar to the query image, documents similar to the query document, or proteins similar to the query protein.

[0059] Figure 3The embedded process 300 in an implementation of the enhanced similarity search is illustrated. In some implementations, the embedded process 300 can be employed in the context of the search process 200. In addition, the embedded process 300 can be implemented as program instructions in the context of any of hardware, software applications, modules, or other such programming elements comprising a search environment (such as the search environment 100). Figure 3 , the program instructions direct its host computing system(s) to operate as described with respect to the embedding process 300 .

[0060] In operation, embedding process 300 obtains, receives, or otherwise identifies a query object, such as an image, document, protein, or the like. The query object comprises the subject of a similarity search against a dataset comprising other objects. For illustrative purposes, it is assumed that there exists a graph of existing objects in the dataset, and that the graph comprises edges and nodes representing the objects and their connections. The dataset also comprises embeddings for the objects. It is also assumed that the graph and embeddings have been used to train a model capable of generating new embeddings for the query object.

[0061] The embedding process 300 continues by identifying the top k nearest neighbors to the query object in the graph (step 301). In some embodiments, the nearest neighbor search can include a visual similarity search of objects in the dataset based on a neural network, which is different from the model trained for embedding, where the neural network is trained and optimized for learning representations in high-dimensional space. Therefore, similarity can be captured in a distance metric that can be evaluated to find nearest neighbors. Although visual similarity is discussed herein, any type of similarity between two nodes can be evaluated instead, whether between nodes of the same type or between nodes of different types.

[0062] The embedding process 300 also identifies one or more tags associated with the object (step 303). Examples of tags include one or more words that may be part of the object, such as words embedded in an image or included in a document, words in metadata associated with the image or document, or any other type of words that describe or identify the object. The embedding process 300 (or the association process) may also identify the tag(s) by applying a text recognition algorithm or other suitable tool to the object. Alternatively, the object may have been pre-processed or labeled in advance so that its tags are known to the embedding process 300.

[0063] Next, the embedding process 300 selects connections to the graph between the query object and the nodes in the graph based on the modality identified for the similarity search (step 305). The embedding process 300 starts with all object-object connections between the query object and its top k nearest neighbors in the graph, as well as object-label connections between the query object labels in the graph that are similar to the query object description. Depending on the modality selected for the search, some object-label connections may be excluded from the connections. For example, for a purely visual search, all object-label connections may be excluded, while for a highly conceptual search, all object-label connections may be included.

[0064] The embedding process 300 submits the query object and its selected connections to the model to obtain an embedding of the query object (step 307). Submitting the query object to the model can include providing the query object itself to an interface of the model, which can then transform the query object into a vector representation suitable for the model. Similarly, the selected connections can also be submitted via the interface so that the model and / or its environment can transform the connections into vector representations. Alternatively, the embedding process 300 or a subprocess can transform the query object and the selected connections into a vector representation suitable for input to the model.

[0065] The model takes as input the query object and the selected connections and produces as output an embedding (feature vector) of the query object. Because the model has been jointly trained on visual data and conceptual data (labels), the resulting embedding may have both visual and conceptual information encoded in it, the extent of which depends on the selected search modality. Searches that are more visual will give less weight to conceptual information, while searches that are more conceptual will give more weight to conceptual information. This is done in step 307 above, where object-label connections are included or excluded to the extent determined by the modality.

[0066] The embedding process 300 submits the generated embedding to the similarity search component to obtain results including objects in the dataset that are similar to the query object (step 309). The results are returned and may include one or more objects found to be similar to the query object on a visual basis, a conceptual basis, or a combination thereof.

[0067] Figures 4A to 4B Two different applications of embedding process 300 to elements of search environment 100 are illustrated. It will be appreciated that although the steps of embedding process 300 may be attributed to specific components in search environment 100, they may be implemented by other components. Figures 4A to 4B One or more of the components illustrated, as well as other components not shown or included in search environment 100 , perform.

[0068] Figure 4A An exemplary search scenario is illustrated, wherein the selected modality includes a visual modality, and Figure 4B The concepts illustrated and discussed below are modal opposites. Figure 4A In

[0045] , model training component 101 obtains graph 107 of existing objects in dataset 105. As mentioned, graph 107 includes nodes and edges corresponding to objects and labels in the dataset, and their interconnections. In this example, label nodes in graph 107 are represented by label nodes that are filled darker than other nodes, while object nodes are represented by object nodes that are filled with a relatively lighter color. Model training component 101 trains model 110 on the graph and the embeddings of the objects in the graph, so that model 110 can be provided in its trained state embedding component 111.

[0069] A similarity search is then initiated with respect to the query object 113. The modality selector 112 determines the modality for the search, which informs the embedding component 111 how to generate an embedding for the query object 113. The modality can be determined explicitly, implicitly, or otherwise, from a set of possible modalities, a range, or other set. For exemplary purposes, assume that the query object 113 is an image and the determined modality is a visual modality.

[0070] The embedding component 111 proceeds to generate an embedding for the query object, starting by identifying the top k nearest neighbors to the query object in the graph. The nearest neighbor search can include a visual similarity search for objects in the dataset based on a neural network, although other search methods can be utilized. Although visual similarity is discussed herein, any type of similarity between two nodes can be evaluated instead, whether between nodes of the same type or between nodes of different types. The top k nodes in the graph 107 are highlighted by a rectangle surrounding the node, which rectangle includes both object nodes. The embedding component 111 also identifies one or more labels associated with the object represented in this example by the label node 135, which is also surrounded by a rectangle.

[0071] Examples of labels include one or more words that may be part of an object, such as words embedded in an image or included in a document, words in metadata associated with an image or document, or any other type of words that describe or identify an object. The embedding process 300 (or association process) can identify the label(s) by applying a text recognition algorithm or other suitable tool relative to the object. Alternatively, the object may have been pre-processed or labeled in advance so that its labels are known to the embedding component 111.

[0072] Next, embedding component 111 selectively identifies connections between the query object and other objects in the graph based on the determined modality. Embedding component 111 connects query object 113 to its nearest neighbors in graph 107 based on the visual modality, excluding label nodes. In other words, in this example, the visual modality places less emphasis on label nodes than the conceptual modality, as indicated by the expression tags. mv <tags mc Thus, the node 133 representing the query object 113 in the graph is connected to the two object nodes in the graph via the edge 137, while the label node 135 is excluded from the connected group.

[0073] The query object and its connections, which clearly lack any object-label connections, are then input to the model 110 to produce an output comprising a query embedding 115. Internally, the model 110, because it has also been trained jointly on object-object nodes and object-label nodes, the resulting embedding may already have some object-label information encoded in it. However, this information may be less than if the input included one or more object-label connections.

[0074] In other words, the object-label connection between the query object 113 and the label node in the graph 107 has less influence when no object-label connection is included in the input relative to when one or more such connections are included. This statement necessarily implies that the object-object connection has a greater influence on the resulting embedding when zero object-label connections are included, and has relatively less influence when object-label connections are included in the input. Importantly, the model 110 is able to respond to both extreme cases—when zero object-label connections are included and when many object-label connections are included—because it has been jointly trained on the graph 107. The model 110 is also able to respond to the intermediate case where one or more object-label connections are included in the input data. A technical effect of this flexibility is that any user, upstream application, component, or service has the ability to determine which modality to use on a per-search basis without having to change, swap, or retrain models, or maintain multiple models.

[0075] Once generated, the query embedding 115 is submitted to the evaluation component 121. The evaluation component 121 provides the query embedding 115 as input to the search function 123, which can compare the query embedding with the embedding associated with the dataset 105, whether using a machine learning approach, a rule-based approach, or any other type of search. Given the modality of the search, the results output by the search function 123 are the top k objects that are considered most similar to the query object 113. The results 127 can be, for example, images similar to the query image, documents similar to the query document, video clips similar to the query video, or proteins similar to the query protein.

[0076] exist Figure 4B In the example, model training component 101 again obtains graph 107 of existing objects in dataset 105. Label nodes in graph 107 are represented by label nodes filled darker than other nodes, while object nodes are represented by object nodes filled with a relatively lighter color. Model training component 101 trains model 110 on the embeddings of the graph and the objects in the graph, so that model 110 can be provided in its trained state embedding component 111.

[0077] Then, a similarity search is initiated with respect to the query object 113. The modality selector 112 determines the modality for the search, which informs the embedding component 111 how to generate an embedding for the query object 113. The modality can be determined explicitly, implicitly, or otherwise through a set of possible modalities, ranges, or other. For exemplary purposes, assume that the query object 113 is an image and the modality determined is a concept modality, Figure 4A The visual modes shown are reversed.

[0078] The embedding component 111 proceeds to generate an embedding for the query object, starting by identifying the top k nearest neighbors to the query object in the graph. The nearest neighbor search can include a visual similarity search for objects in the dataset based on a neural network, although other search methods can be utilized. Although visual similarity is discussed herein, any type of similarity between two nodes can be evaluated instead, whether between nodes of the same type or between nodes of different types. The top k nodes in the graph 107 are identified by a rectangle surrounding the node, which rectangle includes both object nodes. The embedding component 111 also identifies one or more labels associated with the object represented in this example by label node 135, which is also surrounded by a rectangle.

[0079] Examples of labels include one or more words that may be part of an object, such as words embedded in an image or included in a document, words in metadata associated with an image or document, or any other type of words that describe or identify an object. The embedding process 300 (or association process) can identify the label(s) by applying a text recognition algorithm or other suitable tool relative to the object. Alternatively, the object may have been pre-processed or labeled in advance so that its labels are known to the embedding component 111.

[0080] Next, the embedding component 111 selectively identifies connections between the query object and other objects in the graph based on the determined modality. The embedding component 111 connects the query object 113 to its nearest neighbor and one or more labeled nodes in the graph 107 based on the conceptual modality. In other words, in this example, the conceptual modality places greater emphasis on the labeled nodes than the visual modality, as shown by the expression tags. mc >tags mv Thus, the node representing the query object 113 in the graph is connected to two object nodes and one label node in the graph via edges 139 .

[0081] The query object and its connections, explicitly including object-label connections, are then input to model 110 to produce an output comprising query embedding 116. Internally, model 110, because it has been trained equally jointly on object-object nodes and object-label nodes, the resulting embedding may have some object-label information encoded therein, and may have more of this information encoded therein than if the input had excluded all object-label connections.

[0082] In other words, the influence of object-label connections between query object 113 and label nodes in graph 107 is greater when at least one object-label connection is included in the input relative to when such connections are not included. This statement necessarily implies that the influence of object-label connections on the resulting embedding is greater when one or more object-label connections are included, and is relatively less influential when fewer (or no) object-label connections are included in the input. As mentioned, model 110 is able to respond to both extreme cases, as well as intermediate cases, because it has been jointly trained on graph 107. As also mentioned, a technical effect of this flexibility is that any user, upstream application, component, or service has the ability to determine which modality to use on a per-search basis without having to change, swap, or retrain models, or maintain multiple models.

[0083] Once generated, the query embedding 116 is submitted to the evaluation component 121. The evaluation component 121 provides the query embedding 115 as input to the search function 123, which can compare the query embedding with the embedding associated with the dataset 105, whether using a machine learning approach, a rule-based approach, or any other type of search. Given the modality of the search, the results output by the search function 123 are the top k objects that are considered most similar to the query object 113. The results 128 can be, for example, images similar to the query image, documents similar to the query document, video clips similar to the query video, or proteins similar to the query protein.

[0084] Figure 5 The embedded process 500 in another implementation of the enhanced similarity search is illustrated. In some implementations, the embedded process 500 can be employed in the context of the search process 200. In addition, the embedded process 500 can be implemented as program instructions in the context of any one of hardware, software applications, modules, or other such programming elements that comprise a search environment such as the search environment 100. Figure 5 , the program instructions direct its host computing system(s) to operate as described with respect to the embedding process 500 .

[0085] In operation, embedding process 500 obtains, receives, or otherwise identifies a query object, such as an image, document, protein, or the like. The query object comprises the subject of a similarity search against a dataset comprising other objects. For illustrative purposes, it is assumed that there exists a graph of existing objects in the dataset, and that the graph comprises edges and nodes representing the objects and their connections. The dataset also comprises embeddings for the objects. It is also assumed that the graph and embeddings have been used to train a model capable of generating new embeddings for the query object.

[0086] The embedding process 500 continues by identifying the top k nearest neighbors to the query object in the graph (step 501). In some embodiments, the nearest neighbor search can include a visual similarity search of objects in the dataset based on a neural network, which is different from the model trained for embedding, where the neural network is trained and optimized for learning representations in high-dimensional space. Therefore, similarity can be captured in a distance metric that can be evaluated to find nearest neighbors. Although visual similarity is discussed herein, any type of similarity between two nodes can be evaluated instead, whether between nodes of the same type or between nodes of different types.

[0087] The embedding process 500 may also identify one or more tags associated with the object. Examples of tags include one or more words that may be part of the object, such as words embedded in an image or included in a document, words in metadata associated with the image or document, or any other type of words that describe or identify the object. The embedding process 500 (or the association process) may identify the tag(s) by applying a text recognition algorithm or other suitable tool to the object. Alternatively, the object may have been pre-processed or labeled in advance so that its tags are known to the embedding process 500.

[0088] Next, the embedding process 500 submits the query object and its selected connections to the model to obtain an embedding of the query object (step 503). Submitting the query object to the model can include providing the query object itself to an interface of the model, which can then transform the query object into a vector representation suitable for the model. Similarly, the selected connections can also be submitted via the interface so that the model and / or its environment can transform the connections into vector representations. Alternatively, the embedding process 500 or a subprocess can transform the query object and the selected connections into a vector representation suitable for input to the model.

[0089] It will be appreciated that the connections may include only object-object connections or a mixture of object-object connections and object-label connections. It will be appreciated that the embedding process 500 may optionally include selecting connections for similarity search based on the determined modality, as described above with respect to Figure 3 For example, for purely visual searches, all object-label connections may be excluded, whereas for conceptual searches, one or more object-label connections may be included.

[0090] The model takes as input a query object and the selected connection and produces an output that includes an initial embedding (feature vector) of the query object. Because the model has been jointly trained on visual data and conceptual data (labels), the initial embedding may have both visual and conceptual information encoded in it, the extent of which depends on whether the input includes object-label connections.

[0091] The embedding process 500 also obtains tag embeddings for the tags associated with the object (step 505). The tag embeddings can be known in advance, or the embedding process 500 can generate tag embeddings at runtime. The embedding process 500 then combines the initial embedding with the tag embedding based on the weighting specified by the modality determined for the search (step 507). Thus, the resulting query embedding is a weighted combination of the initial embedding and the tag embedding. For example, the weighting can influence the combination to encode more tag embeddings into the query embedding for the conceptual modality relative to the visual modality.

[0092] Finally, the embedding process 500 submits the query embedding to the similarity search component to obtain results including objects in the dataset that are similar to the query object (step 509). The results are returned and can include one or more objects found that are similar to the query object on a visual basis, a conceptual basis, or a combination thereof.

[0093] Figure 6A The application of the embedding process 300 in an exemplary search scenario is illustrated, wherein the selected modality includes a visual modality, and Figure 6B The concepts illustrated and discussed below are modal opposites. Figure 6A In

[0045] , model training component 101 obtains graph 107 of existing objects in dataset 105. Graph 107 again includes nodes and edges corresponding to objects and labels in the dataset and their interconnections. Model training component 101 trains model 110 on the graph and the embeddings of the objects in the graph, so that model 110 can be provided in its trained state embedding component 111.

[0094] A similarity search is then initiated with respect to the query object 113. The embedding component 111 proceeds to generate an embedding for the query object, starting by identifying the top k nearest neighbors to the query object in the graph. The nearest neighbor search can include a visual similarity search for objects in a dataset based on a neural network, although other search methods can be utilized. Although visual similarity is discussed herein, any type of similarity between two nodes can be evaluated instead, whether between nodes of the same type or between nodes of different types. The embedding component 111 also identifies one or more tags associated with the object. Examples of tags include one or more words that may be part of the object, such as words embedded in an image or included in a document, words in metadata associated with an image or document, or any other type of words that describe or identify about the object.

[0095] The embedding component 111 identifies connections between the query object and other objects in the graph. In some implementations, the embedding component 111 identifies connections on the same basis regardless of the determined modality, for example by selecting all connections to all top k similar objects. In other implementations, the embedding component 111 may selectively select connections based on the determined modality, as described above with respect to Figure 3 discussed.

[0096] The query object and its connections are then input to the model 110 to produce an output comprising an initial embedding 141. The embedding component 111 combines the initial embedding 141 with the tag embedding 143 based on a modality-specific weighting determined for the search by the modality selector 112. In the case of visual search, the modality gives greater weight to the initial embedding 141 produced by the model 110 than in the context of conceptual search, as expressed by the expression w(t) v<w(t) c In contrast, conceptual search gives greater weight to label embeddings than visual search. Therefore, the resulting combined embedding 145 has a degree of conceptual information encoded therein, which depends on the weighting strength of the label embeddings in the combination. Therefore, the influence of the labels associated with the query object is greater for conceptual search than for visual search. The embedding component 111 is able to respond to both extreme cases (when the weighting of label embeddings is zero, and when the weighting of label embeddings is maximum) as well as an intermediate case when the weighting of label embeddings is moderate. The technical effect of this flexibility is that any user, upstream application, component or service once again has the ability to determine which modality to use on a per-search basis without having to change, swap or retrain models, or maintain multiple models.

[0097] Once generated, the combined embedding 145 is submitted to the evaluation component 121. The evaluation component 121 provides the combined embedding 145 as input to the search function 123, which can compare the combined embedding with the embedding associated with the dataset 105, whether using a machine learning approach, a rule-based approach, or any other type of search. Given the modality of the search, the results output by the search function 123 are the top k objects that are considered most similar to the query object 113. The results 147 can be, for example, images similar to the query image, documents similar to the query document, video clips similar to the query video, or proteins similar to the query protein.

[0098] Figure 6B An exemplary search scenario is illustrated where the selected modality includes a conceptual modality as opposed to a visual modality. Figure 6B In

[0045] , model training component 101 obtains graph 107 of existing objects in dataset 105. Graph 107 again includes nodes and edges corresponding to objects and labels in the dataset and their interconnections. Model training component 101 trains model 110 on the graph and the embeddings of the objects in the graph, so that model 110 can be provided in its trained state embedding component 111.

[0099] A similarity search is then initiated with respect to the query object 113. The embedding component 111 proceeds to generate an embedding for the query object, starting by identifying the top k nearest neighbors to the query object in the graph. The nearest neighbor search can include a visual similarity search for objects in a dataset based on a neural network, although other search methods can be utilized. Although visual similarity is discussed herein, any type of similarity between two nodes can be evaluated instead, whether between nodes of the same type or between nodes of different types. The embedding component 111 also identifies one or more tags associated with the object. Examples of tags include one or more words that may be part of the object, such as words embedded in an image or included in a document, words in metadata associated with an image or document, or any other type of words that describe or identify about the object.

[0100] The embedding component 111 identifies connections between the query object and other objects in the graph. In some implementations, the embedding component 111 identifies connections on the same basis regardless of the determined modality, for example by selecting all connections to all top k similar objects. In other implementations, the embedding component 111 may selectively select connections based on the determined modality, as described above with respect to Figure 3 discussed.

[0101] The query object and its connections are then input to the model 110 to produce an output comprising an initial embedding 141. The embedding component 111 combines the initial embedding 141 with the tag embedding 143 based on a modality-specific weighting determined for the search by the modality selector 112. In the case of concept search, the modality gives less weight to the initial embedding 141 produced by the model 110 than in the context of visual search, as expressed by the expression w(t) c >w(t) v In other words, conceptual search gives greater weight to tag embeddings than visual search. Therefore, the resulting combined embedding 146 has a certain degree of conceptual information encoded in it, which depends on the weighted strength of the tag embeddings in the combination.

[0102] Once generated, the combined embedding 146 is submitted to the evaluation component 121. The evaluation component 121 provides the combined embedding 146 as input to the search function 123, which can compare the combined embedding with the embedding associated with the dataset 105, whether using a machine learning approach, a rule-based approach, or any other type of search. Given the modality of the search, the results output by the search function 123 are the top k objects that are considered most similar to the query object 113. The results 148 can be, for example, images similar to the query image, documents similar to the query document, video clips similar to the query video, or proteins similar to the query protein.

[0103] 7A to 7D The diagram illustrates an operational scenario 700 in a highly simplified example implementation of the enhanced similarity search proposed herein. The operational scenario 700 begins with an existing graph 710 of objects, labels, and their connections. The existing graph 710 includes nodes representing images in a dataset and labels describing a sequence of images. For example, the nodes include image nodes 701, 703, 705, 709, 711, 715, 719, and 721. The nodes also include label nodes 707, 713, and 717.

[0104] In the preprocessing step, some pairs of nodes have been connected by edges based on their similarity and / or applicability. For example, image node 701 is connected to image node 703 via edge 704 because both are images of one or more cats. Subsequently, image node 701 is connected to label node 707 via edge 708, and image node 703 is also connected to label node 707 via edge 706 because label node 707 includes the description "cat."

[0105] Label node 707 is also connected to image node 705 via edge 712 because it also includes a cat in its image. However, image node 705 is not considered sufficiently similar to image node 701 or image node 703 to be connected. Image node 705 is connected to image node 709 via edge 714 because both include an image of a toaster. Image node 709 is connected to label 722 because label 722 includes the description "toaster," even though image node 705 does not. Additionally, although image node 711 is not connected to label node 713, image node 709 is connected to image node 711 via edge 720.

[0106] Image nodes 715 for dogs and cats are connected to both image node 703 and label node 717 via edges 718 and 724, respectively. Label node 717 is also connected to image node 719 and image node 721 via edges 726 and 728, respectively.

[0107] The reasons why some image nodes may be connected to each other but not to other image nodes may include various factors that are generally outside the scope of this disclosure. However, one reason is that the graph can be constructed in different ways based on different judgments or thresholds relative to each other. This leads to different conclusions about which nodes should be connected by edges in different situations. Thus, in this example, image node 701 can be connected to image node 703 but not to image node 705, even though both include cats. Similarly, with respect to image nodes 715, 721, and 719, even though all three include images of dogs.

[0108] Likewise, many factors may influence why some image nodes are connected to label nodes while others are not, or why image nodes associated with seemingly similar images are not connected to the same label. For example, image node 711 corresponds to an image of a toaster, but is not connected to label node 713. This may be because the image corresponding to image node 711 does not have textual data or other such information to allow it to be identified as a toaster or connected to a label.

[0109] Thus, it can be understood that most graphs do not (and cannot) perfectly represent the interconnections between objects and labels in a dataset. Instead, based on various factors and thresholds that determine whether a node should be connected to another node, an almost infinite number of connections are possible. This reality is precisely why the enhanced similarity search disclosed herein provides an advantage over the prior art - because it allows the connections in the graph to be jointly represented in the model, which then allows the model to be selectively exploited in the direction of one modality or the other.

[0110] Continue to operate scene 700, Figure 7B A query image 731 is provided in

[15] . Query image 731 is ostensibly a cat, and the goal of the similarity search is to find the top k most similar images in the existing graph 710 based on the selected modality. The first step is to identify the nearest neighbors to query image 731 in the existing graph 710 based on a separate similarity search. The search results in image nodes 701 and 703 as the two most similar to query image 731.

[0111] Next, in Figure 7C , edges are established between the query image 731 and its nearest neighbors according to the visual modality. According to the visual modality in this example, zero object-label connections are included. Therefore, when an embedding is generated for the query image, the query image 731 only includes edge 732 and edge 736. The embedding is generated by a machine learning model (e.g., model 110) with the query image 731 and its edges 732 and 736 as input. The resulting query embedding is then supplied to an evaluation component to identify the most similar image in the graph relative to the query image and based on its embedding. Search result 750 includes image node 701 and image node 715 that are most similar to the query image 731 on a visual basis.

[0112] In contrast, Figure 7D The diagram shows an example of a conceptual modality. Figure 7D, edges are established between the query image 731 and its nearest neighbors based on the visual similarity between them. However, according to the conceptual modality in this example, an edge 734 of the label node 707 is also included. Therefore, when an embedding is generated for the query image, the query image 731 includes edges 736, 732, and edge 734. The embedding is generated by a machine learning model (e.g., model 110) with the query image 731 and its edges 732, 734, and 736 as input. The resulting query embedding is then supplied to the evaluation component to identify the most similar image in the graph relative to the query image and based on its embedding. Search result 755 includes image node 701 and image node 705 that are most similar to the query image 731 on a visual basis.

[0113] It is worth noting that search result 755 is different from search result 750 because the input to the embedding process is different. Figure 7C In , the input lacks object-label edges, while in Figure 7D , the input includes object-label edges. In one set (search result 750), the image of a dog associated with image node 715 may be more visually similar to query image 731 than the image of a cat and toaster associated with image node 705. In another set (search result 755), the image of a cat and toaster replaces the image of a dog, which may be due to its connection to label node 707. In other words, although the image of a dog and a cat curled up together (image node 715) looks more similar to computer vision visually than the image of a cat and a toaster, the images of cats and toasters are more conceptually similar. The enhanced similarity search and unified framework disclosed herein allow users to explore these different results without the burden of having to retrain, maintain, or replace multiple models.

[0114] Figure 8 The diagram shows a computing system 801 that represents any system or collection of systems that can implement the various components, modules, processes, procedures, and scenarios disclosed herein. Examples of computing system 801 include, but are not limited to, server computers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machines, containers, and any variations or combinations thereof. Other examples include desktop computers, laptop computers, tablet computers, Internet of Things (IoT) devices, wearable devices, and any other physical or virtual combinations or variations thereof.

[0115] The computing system 801 can be implemented as a single device, system, or apparatus, or can be implemented as multiple devices, systems, or apparatuses in a distributed manner. The computing system 801 includes, but is not limited to, a processing system 802, a storage system 803, software 805, a communication interface system 807, and a user interface system 809 (optional). The processing system 802 is operatively coupled to the storage system 803, the communication interface system 807, and the user interface system 809.

[0116] Processing system 802 loads and executes software 805 from storage system 803. Software 805 includes and implements a search process 806, which is representative of the search process discussed with respect to the previous figures. Software 805 also includes and implements an embedding process 816, which is representative of the embedding process discussed with respect to the previous figures. When executed by processing system 802 to provide enhanced similarity search, software 805 directs processing system 802 to operate as described herein with respect to the various processes, operating scenarios, and sequences discussed in at least the aforementioned implementations. Computing system 801 may optionally include additional devices, features, or functionality not discussed for the sake of brevity.

[0117] Still refer to Figure 8 , processing system 802 may include a microprocessor and other circuitry that retrieves and executes software 805 from a storage system 803. Processing system 802 may be implemented within a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of processing system 802 include general-purpose central processing units, graphics processing units, special-purpose processors and logic devices, and any other type of processing device, combination, or variation thereof.

[0118] The storage system 803 may include any computer-readable storage medium that is readable by the processing system 802 and capable of storing the software 805. The storage system 803 may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, flash memory, virtual and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage medium. In any case, a computer-readable storage medium is not a propagated signal.

[0119] In addition to computer-readable storage media, in some implementations, storage system 803 may also include computer-readable communication media over which at least some of software 805 may be transferred internally or externally. Storage system 803 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems that are co-located or distributed relative to each other. Storage system 803 may include additional elements, such as a controller, that can communicate with processing system 802 or possibly other systems.

[0120] The software 805 (including the search process 806 and the embedding process 816) can be implemented in program instructions and, among other functions, when executed by the processing system 802, can direct the computing system 801 to operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, the software 805 can include program instructions for implementing the enhanced similarity search described herein.

[0121] In some embodiments, the software 805 may include a plurality of program instructions, such as a program code or a program code that is used to execute the program. In some embodiments, the software 805 may include a plurality of program instructions, such as a program code or a program code that is used to execute the program. In some embodiments, the software 805 may include a plurality of program instructions, such as a program code or a program code that is used to execute the program. In some embodiments, the software 805 may include a plurality of program instructions, such as a program code or a program code that is used to execute the program. In some embodiments, the software 805 may include a plurality of program instructions, such as a program code or a program code that is used to execute the program. In some embodiments, the software 805 may include a plurality of program instructions, such as a program code or a program code that is used to execute the program. In some embodiments, the software 805 may include a plurality of program instructions, such as a program code or a program code that is used to execute the program. In some embodiments, the software 805 may include a plurality of program instructions, such as a program code or a program code that is used to execute the program.

[0122] Typically, when software 805 is loaded into processing system 802 and executed, the software 805 can transform a generally suitable device, system, or apparatus (represented by computing system 801) from a general computing system to a specialized computing system customized to provide enhanced similarity search. In practice, the coded software 805 on storage system 803 can transform the physical structure of storage system 803. The specific transformation of the physical structure can depend on various factors in different implementations of this description. Examples of such factors can include, but are not limited to, the technology of the storage medium used to implement storage system 803, whether the computer storage medium is characterized as a primary storage device or a secondary storage device, and other factors.

[0123] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, then when program instructions are encoded therein, the software 805 can transform the physical state of the semiconductor memory, such as by transforming the state of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. Similar transformations may occur with respect to magnetic or optical media. Other transformations of the physical medium are also possible without departing from the scope of this description, wherein the foregoing examples are provided merely to facilitate this discussion.

[0124] The communication interface system 807 may include communication connections and devices that allow communication with other computing systems (not shown) via a communication network (not shown). Examples of connections and devices that together allow inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The connections and devices may communicate over a communication medium, such as metal, glass, air, or any other suitable communication medium, to exchange communications with other computing systems or systems. The aforementioned media, connections, and devices are well known and need not be discussed in detail herein.

[0125] Communications between computing system 801 and other computing systems (not shown) can occur over one or more communication networks and according to various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, the internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variations thereof. The aforementioned communication networks and protocols are well known and need not be discussed in detail here.

[0126] The included description and drawings depict specific embodiments to teach those skilled in the art how to make and use the best mode. In order to teach the principles of the invention, some conventional aspects have been simplified or omitted. Those skilled in the art will appreciate variations from these embodiments that fall within the scope of the present disclosure. Those skilled in the art will also appreciate that the above-described features can be combined in various ways to form multiple embodiments. Therefore, the present invention is not limited to the specific embodiments or examples described above, but is limited only by the claims and their equivalents.

Claims

1. A computer-implemented method executed by one or more hardware processors, the method comprising: identifying, from a plurality of search modalities, a selected search modality for a similarity search of a database of objects; identifying new edges between a query object and a subset of nodes selected from at least one of a first node type or a second node type based on the selected search modality using a graph, the graph representing a database of the objects and jointly modeling the plurality of search modalities using nodes, the nodes of the first node type representing the objects, the nodes of the second node type representing labels of the objects, edges connecting similar nodes, wherein the edges include object-object connections between a first set of nodes representing corresponding objects, and object-label connections between a second set of nodes representing connections between at least some of the objects and corresponding labels; using a neural network trained on the plurality of search modalities to convert the query object into a query object embedding by executing a neural network on the query object and the new edge, such that the query object embedding is biased toward the selected search modality, wherein the neural network includes neurons associated with weights, and when the neural network is trained, the weights change based on a difference between a result generated by the neural network and a correct embedding of an input to the neural network; as well as The similarity search of the database of objects is performed based on similarity to the query object embedding, the similarity search identifying one or more similar objects to the object.

2. The method according to claim 1, wherein The edges include object-object connections between the first set of nodes of the first node type representing corresponding similar objects, and object-label connections between the second set of nodes of the second node type representing connections between at least some of the objects and corresponding labels. 3 . The method of claim 1 , further comprising determining whether to select the subset of nodes from the first node type or the second node type based on the select search modality.

4. The method of claim 1 , further comprising identifying the new edge between the query object and the subset of nodes in the graph based at least on: Identifying the top k subsets of the objects based on their similarity to the query object; identifying one or more of the tags associated with the query object; as well as The new edge includes at least one of the following: a new object-object connection between the query object and one of the top k subsets of objects, or a new object-label connection between the query object and one of the one or more labels associated with the query object.

5. The method according to claim 1, wherein The selective search modality is a visual modality, and the method further includes including a new object-object connection between the query object and one of the objects in the new edge, and excluding an identified object-label connection between the query object and one of the labels represented by the graph from the new edge.

6. The method according to claim 1, wherein The selective search modality is a concept modality, and the method further includes including in the new edge: (i) a new object-object connection between the query object and one of the objects, and (ii) an identified object-label connection between the query object and one of the labels represented by the graph.

7. The method of claim 1 , wherein identifying the selected search modality comprises: User input is received identifying the selected search modality from the plurality of search modalities including a visual modality and a conceptual modality.

8. The method according to claim 1, wherein The query object comprises a query image, the object comprises an image represented by the node in the graph, and the one or more similar objects identified by the similarity search comprise k nearest neighbors of the query image in the image.

9. One or more computer storage media storing computer-usable instructions that, when used by one or more hardware processors of one or more computing devices, cause the one or more computing devices to perform operations comprising: identifying, from a plurality of search modalities, a selected search modality for a similarity search of a database of objects; identifying new edges between a query object and a subset of nodes selected from at least one of a first node type or a second node type based on the selected search modality using a graph, the graph representing a database of the objects and jointly modeling the plurality of search modalities using nodes, the nodes of the first node type representing the objects, the nodes of the second node type representing labels of the objects, edges connecting similar nodes, wherein the edges include object-object connections between a first set of nodes representing corresponding objects, and object-label connections between a second set of nodes representing connections between at least some of the objects and corresponding labels; using a neural network trained on the plurality of search modalities to convert the query object into a query object embedding by executing a neural network on the query object and the new edge, such that the query object embedding is biased toward the selected search modality, wherein the neural network includes neurons associated with weights, and when the neural network is trained, the weights change based on a difference between a result generated by the neural network and a correct embedding of an input to the neural network; as well as The similarity search of the database of objects is performed based on similarity to the query object embedding, the similarity search identifying one or more similar objects to the object.

10. The one or more computer storage media of claim 9, the operations further comprising determining whether to select the subset of nodes from the first node type or the second node type based on the select search modality.

11. The one or more computer storage media of claim 9, the operations further comprising identifying the new edges between the query object and the subset of nodes in the graph based at least on: Identifying the top k subsets of the objects based on their similarity to the query object; identifying one or more of the tags associated with the query object; as well as The new edge includes at least one of the following: a new object-object connection between the query object and one of the top k subsets of objects, or a new object-label connection between the query object and one of the one or more labels associated with the query object.

12. One or more computer storage media according to claim 9, wherein: The selective search modality is a visual modality, and the operation further includes including a new object-object connection between the query object and one of the objects in the new edge, and excluding an identified object-label connection between the query object and one of the labels represented by the graph from the new edge.

13. A computer system comprising one or more hardware processors and a memory, the memory being configured to provide computer program instructions to the one or more hardware processors, and when the computer program instructions are executed by the one or more hardware processors, causing the one or more hardware processors to perform operations comprising: identifying, from a plurality of search modalities, a selected search modality for a similarity search of a database of objects; identifying new edges between a query object and a subset of nodes selected from at least one of a first node type or a second node type based on the selected search modality using a graph, the graph representing a database of the objects and jointly modeling the plurality of search modalities using nodes, the nodes of the first node type representing the objects, the nodes of the second node type representing labels of the objects, edges connecting similar nodes, wherein the edges include object-object connections representing corresponding objects between a first set of nodes, and object-label connections between a second set of nodes representing connections between at least some of the objects and corresponding labels; using a neural network trained on the plurality of search modalities to convert the query object into a query object embedding by executing a neural network on the query object and the new edge, such that the query object embedding is biased toward the selected search modality, wherein the neural network includes neurons associated with weights, and when the neural network is trained, the weights change based on a difference between a result generated by the neural network and a correct embedding of an input to the neural network; as well as The similarity search of the database of objects is performed based on similarity to the query object embedding, the similarity search identifying one or more similar objects to the object.

14. The computer system according to claim 13, wherein: The selected search modality is a concept modality, and the embedding component is further configured to include in the new edge: (i) a new object-object connection between the query object and one of the objects, and (ii) an identified object-label connection between the query object and the label represented by the graph.

15. The computer system according to claim 13, wherein: The operations further include determining whether to select the subset of nodes from the first node type or the second node type based on the select search modality.

16. The computer system of claim 13, the selected search modality is identified based on receiving user input identifying the selected search modality from the plurality of search modalities including a visual modality and a conceptual modality.

17. The computer system according to claim 13, wherein: The query object comprises a query image, the object comprises an image represented by the node in the graph, and the one or more similar objects identified by the similarity search comprise k nearest neighbors of the query image in the image.

18. The computer system according to claim 13, wherein: The edges include object-object connections between the first set of nodes of the first node type representing corresponding similar objects, and object-label connections between the second set of nodes of the second node type representing connections between at least some of the objects and corresponding labels.

19. The computer system of claim 13, the embedding component further configured to identify the new edge between the query object and the subset of nodes in the graph based at least on: Identifying the top k subsets of the objects based on their similarity to the query object; identifying one or more of the tags associated with the query object; as well as The new edge includes at least one of the following: a new object-object connection between the query object and one of the top k subsets of objects, or a new object-label connection between the query object and one of the one or more labels associated with the query object.

20. The computer system of claim 13, wherein: The selective search modality is a visual modality, and the embedding component is further configured to include a new object-object connection between the query object and one of the objects in the new edge, and to exclude from the new edge an object-label connection identified between the query object and one of the labels represented by the graph.

Images