Computer system and method for automated image retrieval

Abstract

An image retrieval system receives images from an image repository to identify related images therefor. Related images may have the same environment or object as well as features and other characteristics. In an image retrieval graph, images in the repository are represented by a set of image nodes connected by edges to other related image nodes, and the image nodes have edge weights representing the similarity of the nodes to each other. Based on the received images, an image traversal system identifies the images in the image retrieval graph and alternately explores and traverses (also referred to as "exploiting") image nodes with edge weights. In the exploration step, image nodes in the exploration set are evaluated to identify connected nodes that are added to the traversal set of the image nodes. In the traversal step, related nodes in the traversal set are added to the exploration set and the query result set.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 770,159, filed on November 20, 2018, which is incorporated by reference in its entirety. Background Art

[0003] The present invention generally relates to retrieving images related to a query image, and more particularly to identifying related images by exploring and traversing an image retrieval graph.

[0004] Identifying related images by a computing system is a challenging problem. From a query image used as input, an image retrieval system attempts to identify related images that are similar or related to the query image. Related images share features with each other and may, for example, be different images taken of the same scene, environment, object, etc. Despite sharing this similarity, images may differ in many ways even if taken in the same context. Illumination, perspective, background clutter, and other characteristics of image capture may change the way a given environment appears in an image, increasing the challenge of identifying these related images. For example, four different images of the Eiffel Tower may differ in angle, lighting, and other characteristics. Using one of these images to identify the other three in a repository of thousands or millions of other images is challenging. As a result, the success rate of successfully identifying related images (and excluding irrelevant images) in existing methods continues to need improvement. In addition, many methods may also be computationally intensive and require a lot of analysis when the image retrieval system receives a query image, which prevents these methods from running effectively in a real-time execution environment. Summary of the invention

[0005] The image retrieval system uses an image retrieval graph to identify images related to the image specified in the query. The image retrieval graph represents images as image nodes connected by edges in the graph. The edges have edge weights that represent the similarity of the two images. The image retrieval graph in one embodiment is generated by identifying the related images to be connected in the graph, and then identifying the weights for the edges connecting the related images in the image retrieval graph. Each image can be represented by an image descriptor that characterizes the image as a vector. In order to determine the related images, the image descriptors between the given image and other images are compared to determine the similarity score between the images based on the similarity between the descriptors. This similarity can be determined by the inner product between the image descriptors. The similarity between different images can be determined, and highly similar images can be selected for generating edges in the graph. In one embodiment, this represents k-NN selection: select the highest k images with the closest neighbors (as measured by the similarity score) of the selected number. After selecting the image connected to the edge, the edge can be assigned a weight based on the determined similarity score, or the weight can be determined based on a weight function, such as a function that compares images based on inlier counts. By identifying which image nodes are connected with edges and weighting the edges accordingly, an image retrieval graph is generated that connects various images in the image repository.

[0006] To search for relevant images in an image repository, an image retrieval system receives a request to identify an image for which to identify similar images in the image repository. To identify relevant images, the image retrieval system searches an image retrieval graph. While performing the search, nodes encountered in the image retrieval graph are assigned to a query result set (relevant results for the query), an exploration set (image nodes to be explored), and a traversal set (image nodes for relevance evaluation and image nodes added to the query result set and the exploration set).

[0007] To perform a search according to one embodiment, the image retrieval system alternates between exploring nodes in the exploration set and traversing ("exploiting") nodes in the traversal set to add nodes from the traversal set to the exploration set (and the query result set) when the relevance of the nodes in the traversal set exceeds a threshold. Each iteration of alternating between these steps is called an explore-exploit iteration. The explore step identifies image nodes in the image retrieval graph that may not be directly connected to the query image, and allows the search to reach image nodes that are far from the query image but are related to the query image. Therefore, the traverse / exploit step uses the direct neighborhood of each image node identified as relevant to the query, and identifies subsequent nodes to add to the query results and subsequently explore.

[0008] Initially, image nodes for images associated with a query are added to an exploration set. When performing an exploration-exploitation iteration, the exploration set is iteratively evaluated to identify nodes connected to nodes in the exploration set, and the traversal set is updated by adding the node to the traversal set. In the traversal set, nodes are stored together with the weights of the edges connecting the nodes in the traversal set to the connected nodes. When explored, the highest weight associated with the node is stored in the traversal set and is called the traversal weight. When an image node in the exploration set is explored (i.e., its connected nodes are added / updated in the traversal set), the node is removed from the exploration set. When the exploration set is empty, the traversal set is evaluated to add additional nodes to the query result set and the exploration set. The traversal weight of the node is compared with a relevance threshold, and when the traversal weight of the image node is higher than the threshold, the image node is added to the exploration set and the query result set. In some embodiments, the traversal set is organized as a prioritized heap of traversal weights. In these embodiments, when the traversal weight is higher than the relevance threshold, the system can read the top of the heap and pop up the image node from the heap. When no nodes remain in the traversal set above the relevance threshold, another iteration of the explore-exploit iteration begins by exploring the newly added image nodes in the exploration set. In some embodiments, to ensure that additional query results continue to be identified, when the exploration set is empty and no image nodes in the traversal set are above the threshold, one or more image nodes with the highest traversal weights are added to the query result set and the exploration set. This ensures that there is always at least one node to explore. In this embodiment, the addition of (multiple) nodes with the highest traversal weight (even if not above the threshold) can also be always performed or continue until the minimum number of minimum explore-exploit iterations is performed.

[0009] The explore-exploit iteration may end when a stopping condition occurs, such as when the traversal adds no additional nodes to the exploration set or when the number of query results exceeds a limit on query results.

[0010] By searching the image retrieval graph in this manner, relevant images can be identified with limited execution time based on traversal of the generated image retrieval graph. For example, this can allow an image retrieval system to perform a search without comparing or modifying the underlying image descriptors to search the image retrieval graph or evaluating similarity at runtime. When spatial verification is used to refine edge weights, exploration can also avoid topic drift during the search by reducing false positives that can occur when comparing vector descriptors between images. The search also therefore identifies images that are still "close" to the query image because the weights are verified by the spatial verification analysis, and the query results can be sorted according to when nodes with sufficient relevance are encountered in the image retrieval graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An example image retrieval system according to one embodiment is shown.

[0012] Figure 2A An example process for generating edges between image nodes of an image retrieval graph is shown according to one embodiment.

[0013] Figure 2B Example similarity scores and generated edges for an image retrieval graph are shown.

[0014] Figure 3 A process for performing image queries on an image retrieval graph according to one embodiment is shown.

[0015] Figure 4A -H shows an example query execution for an example image retrieval graph and the corresponding states of the query result set, exploration set, and traversal set.

[0016] The accompanying drawings depict various embodiments of the present invention for purposes of illustration only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the present invention described herein. DETAILED DESCRIPTION

[0017] Figure 1 An example image retrieval system 100 is shown according to one embodiment. The image retrieval system identifies relevant images based on images associated with a received image query. The image query may be received from another device or may be entered by a user at the image retrieval system 100. The image retrieval system 100 includes various modules and data stores for describing images and returning relevant images in response to a query. These include a descriptor extraction module 110, a graph generation module 120, an image search module 130, an image repository 140, and an image retrieval graph 150. In various embodiments, the image retrieval system 100 may include additional computational modules or data stores. Figure 1 Additional modules and devices that may be included in various embodiments are shown in order to avoid overly complex descriptions. For example, the user device providing the image query and the network through which the device communicates with the image retrieval system 100 are not shown in FIG. Figure 1 Shown in.

[0018] To prepare images for use with a query, relationships between related images are characterized by nodes in an image retrieval graph 150 with weights, the weights between image nodes describing the similarities between images. To retrieve images related to an image query, the image retrieval system 100 identifies image nodes for the query image and performs a search on the image retrieval graph 150 based on the weights between image nodes. In particular, the search alternates between 1) selecting relevant nodes to be added to the query results (referred to as traversal or "exploitation"), and 2) exploring the relevance of additional nodes connected to the selected nodes. This allows the search to be performed with valid results in a relatively short time, which scales effectively with the number of images considered in the search. .

[0019] The image query received by the image retrieval system 100 is associated with an image or otherwise specifies an image. The image retrieval system 100 identifies relevant images for the query image in the image repository 140. The image repository 140 may include thousands or millions of images of various environments and objects that may be considered relevant results for the query. For example, the image repository 140 may include images captured by a vehicle during a trip, or may be images captured by an individual user and uploaded to the image repository 140. Images may be considered related when they have the same person, landmark, scene, or other objects in the image field of view. A particular image may be related to more than one other image. Some images in the image repository 140 may be labeled with a relationship to other images, such as identifying that two images each show the same object. These images may be used as a training set, for example, for training the descriptor model discussed with respect to the descriptor extraction module 110. Images are typically represented in various color spaces, such as red-blue-green (RGB) or hue-saturation-value (HSV). Images in the image repository may be significantly different in appearance from each other, even if they are related images. The various images may vary significantly with respect to brightness, color, field of view, perspective, additional objects or environments captured in the image, and so forth.

[0020] The descriptor extraction module 110 extracts image descriptors from images in the image repository. An image descriptor is a calculated description of an image that can be used to compare images in the image retrieval system 100. An image descriptor can include a vector representation of an image that provides a compact representation that can be compared across images to determine the similarity between the images as a whole. In one embodiment, the vector representation is a vector of 2048 values. An image descriptor can also include feature descriptors that describe "interesting" portions of an image, such as edges or other features detected based on the image. Feature descriptors can be used to identify matching portions of two images. Therefore, while the vector representation can be a general, global feature of an image, a feature descriptor can describe a single feature, point, or portion of an image that can be compared to determine whether the images match.

[0021] To generate a vector representation of an image, the descriptor extraction module 110 can apply a neural network that is trained to receive an image and output a vector representation of the image. The image retrieval system 100 can use a pre-trained neural network that outputs a vector representation of an image, or train the images based on a training set of images with labeled relationships in the image repository 140. The similarity scores between the images are used to train the neural network as a goal of the network so that the similarity scores are compared with the labeled relationships between the images, and the network is trained based on the error between the predicted similarities and the labeled relationships. In one embodiment, the neural network is a fine-tuned CNN-based R-MAC descriptor model for landmark retrieval. In other embodiments, the vector representation can be generated based on a model using local invariant features or a bag of words model.

[0022] The feature descriptors may be specific to a particular location within an image so that taking different images of a given object generates feature descriptors that match across each image. In one embodiment, the feature descriptors are generated based on a scale-invariant feature transform (SIFT). In another embodiment, the feature descriptors are generated based on a deep local feature model (DELF), which may use a convolutional neural network to generate features that describe portions of an image. In one embodiment, the deep local feature model extracts a feature vector from an image and may reduce the dimensionality of the vector based on principal component analysis (PCA).

[0023] The graph generation module 120 uses the image descriptors to generate an image retrieval graph. The graph generation module 120 evaluates the image descriptors to identify similar images and generates edge weights between image nodes.

[0024] Figure 2A An example process for generating edges between image nodes of an image retrieval graph is shown according to one embodiment. Figure 2BAn example similarity score and generated edges for an image retrieval graph are shown. The process may be performed by a descriptor extraction module 110 and a graph generation module 120. Initially, an image descriptor is extracted 200 from each image in an image repository, which may be performed by the descriptor extraction module 110 as described above. In some examples, the image descriptor for an image may be pre-generated or may already be associated with an image in an image repository. Next, the graph generation module 120 evaluates each image to identify which other images are connected to the image in the graph and to identify what weights are assigned to the edges in the graph. In order to identify the images to be connected in the graph, the graph generation module 120 evaluates the similarity 210 by determining the similarity score between the subject image and other images in the image repository. In one embodiment, the similarity score is the inner product between the vector representation of the subject image and the vector representation of another image. The inner product may be a dot product, or may be another vector similarity function for comparing the similarity between vectors. As a working example, a similarity score table 250 shows example similarity scores calculated for image A relative to other images (including images B, C, D).

[0025] Next, the process selects 220 images based on the similarity score to connect to the subject image. The similarity scores are ranked, and the top k images are selected from the ranked list. The number k of selected images may vary in different implementations and may be 50, 100, 200 or more. In the present embodiment, the selected images represent the nearest neighbors (k-NN) to the subject image, as measured by the similarity score. In another embodiment, images above a similarity threshold are selected to be connected in an image retrieval graph, which may include more or less than k. In another embodiment, two schemes may be used to connect at least the top k nearest neighbors and any additional images that exceed the threshold score. In this example, this ensures that at least k image nodes are connected, while also allowing additional connections when more images are highly similar. As a result, the image retrieval graph may be sparse, including k-NN connections for each image node. For example, in a 1M image repository with a k value of 100, each image node is connected to 100 other nodes in the image retrieval graph.

[0026] Next, weights may be determined 230 for each pair of images to be connected. In one embodiment, the weights are similarity scores as described above. In other embodiments, the weights may be determined based on a comparison of feature descriptors of the images. In particular, feature descriptors may be compared to identify features that may match between the two images. The analysis may primarily or exclusively evaluate inliers between the two images, indicating features that match between the two images, even when additional features do not match. As an example, the comparison may be based on a random sample consensus ("RANSAC") algorithm. The algorithm may apply various image transformations to attempt to identify feature matches across images and select the transformation that allows the maximum number of features to match. When comparing feature descriptors of the images, the number of inliers may be used as weights between images in an image retrieval graph. By using an inlier-based analysis to weight edges between image nodes, edges between nodes may therefore verify actual similarity between images based on a description of portions of the image at a finer scale than a global description, which may be determined from a similarity score determined by a vector representation.

[0027] After determining the weights between the image nodes to be connected, edges between the image nodes in the determined weight graph may be generated 240. In one embodiment, the connections and weights between the image nodes may be represented as a sparse matrix, where a value of zero (or no value) at the intersection of two image nodes indicates that there is no edge between the image nodes, and a non-zero value is the weight of the edge between the nodes at that location.

[0028] In one embodiment, the local image retrieval graph 260 shows the connection between the image nodes representing image A and the corresponding weights. In this example, three image nodes are selected for image A, for example, if k is 3 in the k-NN scheme selected for the connected nodes. After weighting the nodes, the weights shown in the local image retrieval graph 260 can be generated. In this example, although the nodes are selected based on the similarity scores as shown in the example similarity score table 250, the weights assigned to the nodes are generated based on the identified inliers between the images. In this example, the inlier score between images A and C is 87, the inlier score between A and B is 80, and the inlier score between A and D is 15. This example illustrates that although images A and D have a similarity score high enough to include an edge between A and D, a relatively low inlier score is generated between A and D based on the verification of the comparison of feature descriptors. This process can be repeated for each image in the image to be added to the image retrieval graph 150 to select 220 nodes and generate 240 edges between them.

[0029] Figure 3The process for performing an image query on an image retrieval graph according to one embodiment is illustrated. The process can be performed by an image search module 130 with respect to an image retrieval graph 150. To begin the process, the image search module 130 receives 300 a search query indicating which image to identify a related image. The query can include an image, or can specify an image at another location, such as an image in an image repository 140. Next, the image search module 130 identifies whether the image has an associated image node in the image retrieval graph. When the image does not have an associated image node, the image is added 310 to the image retrieval graph. As discussed with respect to FIG. 2, image descriptors can be extracted from the image, similar images for the image can be selected to generate edges, and the edges are generated with weights based on the similarity between the images. Therefore, when an image search for a new image is requested, the new image can be quickly added to the image retrieval graph without requiring modification of image nodes or edges related to other images. In addition, subsequent requests for the image can be processed quickly because the image may already be included in the image retrieval graph.

[0030] To perform a search, the image search module 130 uses image node sets to organize the search. These image node sets may include a query result set, an exploration set, and a traversal set. Initially, each of these sets may be initialized to be empty. To begin the search, the query image node associated with the query image is added 320 to the exploration set. During the execution of the search, the image search module 130 may use an explore-exploit iteration to alternate between exploring edges from image nodes of interest ("explore") and evaluating edges to identify additional nodes of interest ("exploit").

[0031] The query result set stores a set of image nodes for the images to be returned as relevant results for the query. The query result set may store the query results as an ordered table or queue, such that additional items added to the query result set are added to the end of the query result set, and the items in the query result set are also typically arranged in order according to expected relevance. In embodiments where the image graph is traversed according to relevance, the relevance order is a result of the traversal order and no further analysis may be required to determine the result order.

[0032] During the exploration phase, edges connected to nodes in the exploration set are explored to identify the relevance of other nodes based on the connections of nodes of interest in the exploration set. As the edges for each image node in the exploration set are evaluated, the image nodes connected to the image nodes in the exploration set are updated 330 in the traversal set. The traversal set stores a set of image nodes connected to the explored nodes and includes traversal weights associated with the image nodes. The traversal weights in the traversal set can specify the highest edge weights encountered by the query when the node is explored. In this way, the traversal weight represents the relevance of a given image node based on the nodes of interest in the current query exploration. In one embodiment, an image node is popped out of the exploration set and the edges connected to the node are updated in the traversal set.

[0033] Therefore, in order to update the traversal set, a previously unexplored image node (e.g., not in the traversal set) can be added to the traversal set, and the image node has a traversal weight that connects the explored image node to the edge weight of the node of interest. For image nodes that have been previously explored (e.g., in the traversal set), the traversal weight can be updated using the edge weight connected to the currently explored image node. When the edge weight is higher than the traversal weight, the traversal weight can be set to the edge weight, and when the edge weight is lower, the traversal weight can remain unchanged. When an image node is already in the query result set, the traversal set will not be updated because the image node has already been explored.

[0034] In some embodiments, the traversal sets are organized as a heap that prioritizes image nodes according to traversal weight, so that the top item in the heap is the image node with the highest traversal weight. The traversal sets can also be stored in a structure or arranged in order so that the highest traversal weight can be easily identified.

[0035] During the traversal or "exploitation" phase, image nodes in the traversal set are evaluated to determine which are relevant enough to be added to the query result set and the traversal set. Based on the traversal weight of the image node, at least one image node in the traversal set is added 340 to the query set and the exploration set.

[0036] As an example, the traversal weight is compared with a relevance threshold, and image nodes above the threshold are considered to be relevant to the search query and are added 340 to the query result set and the exploration set. In an embodiment where the traversal set is a heap, the traversal phase reads the top traversal value of the traversal set, and when the traversal value exceeds the relevance value, the image node is popped up from the traversal set and added to the query result set and the exploration set. The next image node is read until the top traversal value is not higher than the relevance threshold. Because the image nodes in the traversal set can be evaluated according to the traversal weight, adding the image nodes to the query result set is in the expected order of relevance. In one embodiment, when the traversal node of an image node does not exceed the relevance threshold, the image node can be retained in the traversal set so that each image node evaluated in the query traversal appears in the query result set or the traversal set. In another embodiment, when the traversal set does not have a traversal weight that exceeds the relevance threshold, the traversal set can be emptied at the end of the traversal phase.

[0037] In one embodiment, when no node in the traversal set has a traversal weight exceeding a relevance threshold, at least one node is added from the traversal set to the exploration set to ensure that the query continues to identify query results. In other words, in this embodiment, the traversal phase adds at least a minimum number of image nodes in the traversal set to the query result set and the exploration set, and also includes additional nodes in the traversal set that have a traversal weight above the relevance threshold. The inclusion of these nodes below the relevance threshold can be performed for each exploration-exploitation iteration, and in some embodiments is performed for a limited number of exploration-exploitation iterations or until a minimum number of image nodes are in the query result set.

[0038] Using alternating exploration-exploitation iterations, the search therefore evaluates nodes closer to the query image node and explores the graph when the edge weight is high enough to meet the relevance threshold. As a result, without modifying the image retrieval graph (unless, if necessary, the query image is added 310), the query can be executed, and repeated queries can be executed without runtime comparison of image descriptors. Further, the image node may initially appear to be irrelevant to the initial query (e.g., without an edge or an edge weight below the relevance threshold), and the image node may become relevant as other image nodes with sufficiently high edge weights to the image node are explored (thus placing the traversal weight of the image node above the relevance threshold). Similarly, when the edge weight is based on the inner point count, the relevance threshold can reduce false positives and ensure that there are sufficient relationships between images. This is illustrated for image nodes D and E in the example discussed below. In an additional embodiment, the relevance score can be increased based on the number of exploration-exploitation iterations so that as the query exploration moves further away from the query image, the threshold for continuing the query is increased.

[0039] The explore-exploit iteration may be repeated until a stopping condition is reached, such as the number of query results reaching a maximum, or the exploration set is empty after the traversal phase (eg, no nodes in the traversal set exceed a relevance threshold and no nodes are added to the exploration set).

[0040] Figure 4A -H illustrates an example query execution for an example image retrieval graph 400A-H and the corresponding states of query result sets 410A-H, exploration sets 420A-H, and traversal sets 430A-H. In this example, the threshold for relevance is a traversal weight of 60. In this example, the query image corresponds to image node A, and Figure 4A Addition of a query image node to the exploration set 420A is illustrated. In the image retrieval graphs 400A-400H, nodes included in the exploration set have dashed borders, explored nodes are filled with a diagonal pattern, and edge connections in the traversal set are illustrated with dashed lines.

[0041] exist Figure 4A and 4B In between, the exploration phase is performed to explore edges for the exploration set 420A, which includes the image node A. The edges connected to the image node A are used to update the traversal set so that the following connections are added to the traversal set 430B: a connection from node A to node C with a weight of 87, a connection from node A to node B with a weight of 80, a connection from node A to node D with a weight of 15, as shown in FIG. Figure 4B Next, the traversal phase is Figure 4B and 4C is executed between. During this phase, the traversal set 430B is evaluated relative to the relevance threshold. Initially, image node C is evaluated relative to its traversal weight of 87. This value is above the relevance threshold of 60, so image node C is removed from the traversal set and added to the query result set 410C and the exploration set 420C. Similarly, image node B has a traversal weight of 80, which is above the relevance threshold of 60, so image node B is removed from the traversal set and added to the query result set 410C and the exploration set 420C. When the traversal weight of image node D is evaluated, it has a traversal weight of 15, which is below the threshold relevance, so image node D is retained in the traversal set 430B. Since the traversal set now has no image nodes with traversal weights above the relevance threshold, the traversal phase ends, and the process can evaluate whether to perform another exploration-exploitation iteration. In this case, there are still image nodes to be explored in the exploration set 230C, so another exploration-exploitation iteration is performed.

[0042] from Figures 4C to 4D, another exploration phase is performed, which now explores the nodes identified as relevant from the previous phase. In this exploration phase, image node C is explored and removed from exploration set 420C, followed by image node B, until the exploration set is empty, as shown in exploration set 420D. When exploring image node C, traversal set 430C is updated with an edge from image node C to image node F with a traversal weight of 40, and an edge from image node C to image node D with a traversal weight of 66. Image node C and image node B also have edges connected to image node A, which is not added to the traversal set because image node A is a query image node. When exploring image node B, traversal set 430C is updated with an edge from image node B to image node E with a traversal weight of 55, and an edge from image node B to image node D with a traversal weight of 107. When updating with the edge from image node B to image node D, the ergodic weight may update the ergodic weight for image node D, which is determined from the edge from image node C to image node D (i.e., which has a value of 66). When exploration set 420C is empty, the process proceeds to Figure 4D traversal phase to check the traversal set 430D and proceed to Figure 4E While traversing traversal set 430D, the traversal weight for image node D is 107, which is above the relevance threshold of 60, and image node D is therefore added to query result set 410E and exploration set 420E and removed from traversal set 430E. The remaining image nodes E and F have traversal weights below the relevance threshold, so the traversal phase ends. The process evaluates whether to perform additional explore-exploit iterations, and based on the image node currently in the exploration set (image node D), as Figures 4E-4F Another iteration is performed as shown in .

[0043] from Figures 4E to 4F , an exploration phase is performed to explore the image nodes in exploration set 420E. In this phase, image node D in exploration set 420E is evaluated, which has edges to image nodes A, B, C, E, and G. Image node A is the query image node, and image nodes B and C are already in the query result set (and have been explored previously), so these nodes are not updated in traversal set 420E. The edge from image node D to image node E has an edge weight of 65, and this edge weight is updated in traversal set 430F to replace the previous traversal weight of 55. In order to update the edge weight from image node D to image node G, image node G is added to traversal set 430F, which has a traversal weight of 9. After evaluating the connection to image node D, image node D is removed from exploration set 420E and the exploration set is empty, allowing the process to continue for Figure 4FWhen traversing traversal set 430F, the traversal weight for image node E now exceeds the relevance threshold (with Figure 4D Different), and image node E can be added to query result set 410G and exploration set 420G and removed from traversal set 430G. Because exploration set 420G is not empty, the process decides to perform additional exploration-exploitation iterations. Figure 4G to Figure 4H , the exploration phase evaluates image node E in exploration set 420G. The edges from image node E to image nodes B and D are not updated to traversal set 430G because these image nodes have been previously explored and included in query result set 410G. The edge from image node E to image node H is updated in traversal set 430H and has a traversal weight of 52. After exploring image node E, exploration set 420H is empty and the exploration phase ends. In one embodiment, at the traversal phase, no image node in the traversal set exceeds a relevance threshold of 60. In this embodiment, at the end of this explore-exploit iteration, there is no image node 420H in the exploration set, indicating that no further image nodes in the graph need to be considered sufficiently relevant.

[0044] In other embodiments (not shown), where the traversal phase includes at least one image node from the traversal set (regardless of whether the highest traversal weight exceeds a relevance threshold), image node H may be added to the exploration set and the query result set as having the highest traversal weight in the traversal set 430H. In this embodiment, additional exploration-exploitation iterations explore image node H and possibly image node G, and finally image node F.

[0045] Finally, the images associated with the image nodes in the result query set 410 are returned as a query result set for images associated with the image node A.

[0046] As shown in this example, the explore-exploit iteration allows efficient exploration of the image retrieval graph 400A-H, and preserves the order of relevance in the query result set based on the order in which nodes are encountered when they become sufficiently relevant in the traversal set. Additionally, because nodes can be traversed based on weights, no additional analysis is required to evaluate individual images at runtime of the search query. As a result, the method provides an efficient way to perform image queries with low runtime (particularly for images already present in the image retrieval graph), and to efficiently identify additional images in the repository without significantly deviating from the characteristics of the query image.

[0047] The foregoing description of the embodiments of the present disclosure has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible based on the above disclosure.

[0048] Some parts of this specification sheet describe embodiments of the present disclosure according to algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are usually used by technicians in the field of data processing to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally or logically, it should be understood that these operations are implemented by computer programs or equivalent circuits, microcodes, etc. In addition, without loss of generality, it has been proven that it is convenient to refer to the arrangement of these operations as modules sometimes. The described operations and their associated modules can be implemented in software, firmware, hardware or any combination thereof.

[0049] Any steps, operations, and processes described herein may be implemented or performed by one or more hardware or software modules alone or in combination with other devices. In one embodiment, the computing module may be implemented using a computer program product including a computer-readable medium, the computer product including a computer-readable medium, the computer-readable medium including a computer program code, and the computer program code may be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0050] Embodiments of the present disclosure may also relate to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium, or in any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design for increased computing power.

[0051] Embodiments of the present disclosure also relate to products produced by the computing processes described herein. Such products may include information produced from the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium and includes any embodiment of a computer program product or other data combination described herein.

[0052] Finally, the language used in the specification is selected primarily for readability and instructional purposes, and it may not be selected to describe or limit the subject matter of the present invention. Therefore, the scope of the present invention is not intended to be limited by this detailed description, but by any claims issued based on the application herein. Therefore, the disclosure of the embodiments of the present disclosure is intended to be illustrative, but not to limit the scope of the invention set forth in the following claims.

Claims

1. A computer system for automated image retrieval, the computer system comprising: a processor configured to execute instructions; A non-transitory computer readable medium comprising instructions which, when executed by the processor, cause the processor to: receiving a request from a client device to retrieve an image related to a query image; accessing an image retrieval graph having a set of image nodes representing a set of images and connected by a set of edges having edge weights representing similarities between the images represented by the connected image nodes; adding an image node in the set of image nodes that is associated with the query image to an exploration set of image nodes, the exploration set indicating nodes that have edges to be explored; Initializing a traversal set of image nodes, the traversal set indicating nodes connected to a related query result and associated traversal weights, the traversal weights used to determine whether to explore the image node; Perform exploration-exploitation iterations until a stopping condition occurs. Each iteration consists of: for each image node in the exploration set, identifying an image node in the image retrieval graph that is connected to the image node; adding the connected image node to the traversal set, the connected image node having a traversal weight based on the edge weight connecting the image node and the connected node; and removing the image node from the exploration set; and For at least one image node in the traversal set selected based on the traversal weight, adding the image node to a query result set and the exploration set, and removing the image node from the traversal set; The query result set is returned to the client device. 2 . The computer system of claim 1 , wherein the stopping condition is an empty exploration set or the query result set reaches a maximum number of results.

3. The computer system of claim 1, wherein the image nodes in the image retrieval graph are connected to a subset of image nodes. The computer system of claim 1 , wherein the image retrieval graph is sparsely connected.

5. The computer system of claim 1, wherein the explored set of image nodes is an ordered queue. The computer system of claim 1 , wherein the traversal of image nodes is a heap, the heap prioritized by traversal weights. 7 . The computer system of claim 1 , wherein the image nodes connected in the image retrieval graph are selected based on a ranked number of k-nearest neighbors (k-NN).

8. The computer system of claim 1, wherein the edge weights of the edge sets in the image retrieval graph are based on inlier scores of a Random Sample Consensus (RANSAC) algorithm.

9. A method for automated image retrieval, the method comprising: receiving a request from a client device to retrieve an image related to a query image; accessing an image retrieval graph having a set of image nodes representing a set of images and connected by a set of edges having edge weights representing similarities between the images represented by the connected image nodes; adding an image node in the set of image nodes that is associated with the query image to an exploration set of image nodes, the exploration set indicating nodes that have edges to be explored; Initializing a traversal set of image nodes, the traversal set indicating nodes connected to a related query result and associated traversal weights, the traversal weights used to determine whether to explore the image node; Perform exploration-exploitation iterations until a stopping condition occurs. Each iteration consists of: for each image node in the exploration set, identifying an image node in the image retrieval graph that is connected to the image node; adding the connected image node to the traversal set, the connected image node having a traversal weight based on the edge weight connecting the image node and the connected node; and removing the image node from the exploration set; and For at least one image node in the traversal set selected based on the traversal weight, adding the image node to a query result set and the exploration set, and removing the image node from the traversal set; The query result set is returned to the client device. 10 . The method of claim 9 , wherein the stopping condition is an empty exploration set or the query result set reaches a maximum number of results.

11. The method of claim 9, wherein the image nodes in the image retrieval graph are connected to a subset of image nodes.

12. The method of claim 9, wherein the image retrieval graph is sparsely connected.

13. The method of claim 9, wherein the explored set of image nodes is an ordered queue.

14. The method of claim 9, wherein the traversal of image nodes is a heap, the heap prioritized by traversal weights.

15. The method of claim 9, wherein the image nodes connected in the image retrieval graph are selected based on a ranked number of k-nearest neighbors (k-NN).

16. The method of claim 9, wherein the edge weights of the edge sets in the image retrieval graph are based on inlier scores of a random sample consensus (RANSAC) algorithm.

17. A non-transitory computer readable medium comprising computer program code, which when executed by a processor causes the processor to perform the following steps: receiving a request from a client device to retrieve an image related to a query image; accessing an image retrieval graph having a set of image nodes representing a set of images and connected by a set of edges having edge weights consisting of similarities between the images represented by the connected image nodes; adding an image node in the set of image nodes that is associated with the query image to an exploration set of image nodes, the exploration set indicating nodes that have edges to be explored; Initializing a traversal set of image nodes, the traversal set indicating nodes connected to a related query result and associated traversal weights, the traversal weights used to determine whether to explore the image node; Perform exploration-exploitation iterations until a stopping condition occurs. Each iteration consists of: for each image node in the exploration set, identifying an image node connected to the image node in the image retrieval graph; adding the connected image node to the traversal set, the connected image node having a traversal weight based on the edge weight connecting the image node and the connected node; and removing the image node from the exploration set; and For at least one image node in the traversal set selected based on the traversal weight, adding the image node to a query result set and the exploration set, and removing the image node from the traversal set; The query result set is returned to the client device.

18. The non-transitory computer readable medium of claim 17, wherein the traversal of image nodes is a heap, the heap prioritized by traversal weights.

19. The non-transitory computer readable medium of claim 17, wherein the image nodes connected in the image retrieval graph are selected based on a ranked number of k-nearest neighbors (k-NN).

20. The non-transitory computer readable medium of claim 17, wherein the edge weights of the edge sets in the image retrieval graph are based on inlier scores of a Random Sample Consensus (RANSAC) algorithm.

Images